Scribe From Facebook

[Andrew Milkowski]

Hey there!

Want to share some key configurations for Mac when installing/running scribe
with hadoop

For a start these are my environmental parameters:

OS: Mac OS Snow Leopard
Hadoop Version: Cloudera release 0.20.1+152
Thrift: latest svn trunk release
Scribe: git (trunk) 2.0

Best will be if I just quote build files directly (external dependencies
such as libevent, boost etc will manifest themselves during build process in
which case simple mac port install <package> will be required

(Notice use of opt/local/src target destination for build files, it is not a
requirement but nicely localizes build artifacts for possible test/upgrade)

1. hadoop build


ant -verbose package -Dversion="0.20.1" -Dlibhdfs=1 -Dlibrecordio=1
-Dislibhdfs=1 -Dxercescroot=/opt/local/src/xercesc/current
-Djava5.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home
-Dforrest.home=/opt/local/src/apache-forrest/current

Note: for the purpose of scribe integration important ant flags are
-Dlibhdfs=1 and -Dislibhdfs=1 (they build native C++ libraries) 

2. thrift build

./bootstrap.sh
./configure --prefix=/opt/local/src/thrift/trunk/build
JAVA_PREFIX=/opt/local/src/thrift/trunk/build
PY_PREFIX=/opt/local/src/thrift/trunk/build
--exec-prefix=/opt/local/src/thrift/trunk/build
make install

3. scribe build

export
CPPFLAGS="-I/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/include
-I/opt/local/src/hadoop/current/src/c++/libhdfs"
export
LD_LIBRARY_PATH="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Libraries:/opt/local/src/hadoop/current/c++/Mac_OS_X-x86_64-64/lib"
export
LDFLAGS="-L/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Libraries
-L/opt/local/src/hadoop/current/c++/Mac_OS_X-x86_64-64/lib"
export LIBS="-ljvm -lhdfs -framework JavaVM"
export PY_PREFIX="/opt/local/src/scribe/trunk/build"

./bootstrap.sh
./configure --enable-hdfs --prefix=/opt/local/src/scribe/trunk/build
--with-thriftpath=/opt/local/src/thrift/current
--with-fb303path=/opt/local/src/thrift/current/contrib/fb303
-with-fbapth=/opt/local/src/thrift/current/contrib/fb303
make
make install

Note: important flags are: export LIBS="-ljvm -lhdfs -framework JavaVM"
(framework JavaM ie for JNI binding resolution)

Finally start scribe with:

1. scribe test script:

export
DYLD_LIBRARY_PATH="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Libraries:/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/lib"

export
CLASSPATH="/opt/local/src/hadoop/current/hadoop-0.20.1-core.jar:/opt/local/src/hadoop/current/lib/commons-logging-api-1.0.4.jar"


scribed -p 1463 -c${SCRIBE_HOME}/conf/hdfs.conf >>
${SCRIBE_HOME}/logs/scribed.log 2>&1 &

2. test script with (modified version of python script requires appending
import modules because of localization of target build to /opt/local/, this
by itself is not all requirements!)

echo "hello world" | ./scribe_cat test

... beginning of the script scribe_cat
...
...
scribe_catimport sys
sys.path.append("/opt/local/src/scribe/current/lib/python2.6/site-packages")
sys.path.append("/opt/local/src/thrift/current/lib/python2.6/site-packages")
sys.path.append("/opt/local/src/thrift/current/contrib/fb303/lib/python2.6/site-packages")
...
...
end of the script

watch logs streaming into HDFS!

Hope this helps! Mac OS X Snow Leopard, Scribe and Thrift work really great
together!

best regards 


anthony Giardullo wrote:
> 
> 
> 
> Kiran Mudiam wrote:
>> 
>> Has any one had success with compiling Scribe for Mac OS X.
>> 
>> I wish there was a macport for this. How does one go about doing this.
>> 
>> I got the sources and tried compiling and have some dependecy errors.
>> I have installed all the preeqs like libevent, thrift, boost and fb303
>> etc.
>> 
>> 
> 
> I build and run Scribe on my macbook and didn't have any problems after
> installing all the required dependencies.  I would go through building and
> installing the other dependencies again and see if you got any error
> messages you didn't notice previously (especially libevent).  Also, I
> would make sure that you build and install libevent before you build and
> install Thrift.
> 
> And I would direct any future Scribe questions to the Scribe forums at
> https://sourceforge.net/projects/scribeserver/
> 
> -Anthony
> 

-- 
View this message in context: http://old.nabble.com/Scribe-From-Facebook-tp21628575p27348376.html
Sent from the MacPorts - Users mailing list archive at Nabble.com.