From 8503ec1d6bdc3716c764b4508d46b86ceedda73c Mon Sep 17 00:00:00 2001 From: "Joshua E. Jodesty" Date: Thu, 3 Oct 2019 18:50:34 -0400 Subject: [PATCH] semi-final _ --- README.md | 3 ++- distroduce/README.md | 22 +++++++++++++--------- distroduce/dist/distroduce.zip | Bin 36381 -> 36419 bytes 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index b175de4..c611199 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,8 @@ spark-submit --py-files distroduce/dist/distroduce.zip distroduce/local_messagi * [cadCAD Documentation](https://github.com/BlockScience/cadCAD/tree/master/documentation) * [cadCAD Tutorials](https://github.com/BlockScience/cadCAD/tree/master/tutorials) * **Terminology:** - * ***[Initial Conditions](https://github.com/BlockScience/cadCAD/tree/master/documentation#state-variables)*** - State Variables and their initial values (Start event of Simulation) + * ***[Initial Conditions](https://github.com/BlockScience/cadCAD/tree/master/documentation#state-variables)*** - + State Variables and their initial values (Start event of Simulation) ```python initial_conditions = { 'state_variable_1': 0, diff --git a/distroduce/README.md b/distroduce/README.md index b175de4..a5fd47e 100644 --- a/distroduce/README.md +++ b/distroduce/README.md @@ -10,8 +10,8 @@ by Joshua E. Jodesty ``` ## What?: *Description* -***Distributed Produce*** (**[distroduce](distroduce)**) is a message simulation and throughput benchmarking framework / -[cadCAD](https://cadcad.org) execution mode that leverages [Apache Spark](https://spark.apache.org/) and +***Distributed Produce*** (**[distroduce](distroduce)**) is a distributed message simulation and throughput benchmarking +framework / [cadCAD](https://cadcad.org) execution mode that leverages [Apache Spark](https://spark.apache.org/) and [Apache Kafka Producer](https://kafka.apache.org/documentation/#producerapi) for optimizing Kafka cluster configurations and debugging real-time data transformations. *distroduce* leverages cadCAD's user-defined event simulation template and framework to simulate messages sent to Kafka clusters. This enables rapid and iterative design, debugging, and message @@ -20,8 +20,8 @@ Streaming. ##How?: *A Tail of Two Clusters* ***Distributed Produce*** is a Spark Application used as a cadCAD Execution Mode that distributes Kafka Producers, -message simulation, and message publishing to worker nodes of an EMR cluster. Messages published from these workers are -sent to Kafka topics on a Kafka cluster from a Spark bootstrapped EMR cluster. +message simulation, and message publishing to worker nodes of an [AWS EMR](https://aws.amazon.com/emr/) cluster. +Messages published from these workers are sent to Kafka topics on a Kafka cluster from a Spark bootstrapped EMR cluster. ##Why?: *Use Case* * **IoT Event / Device Simulation:** Competes with *AWS IoT Device Simulator* and *Azure IoT Solution Acceleration: @@ -49,9 +49,9 @@ spark-submit --py-files distroduce/dist/distroduce.zip distroduce/local_messagi ### 1. Write cadCAD Simulation: * **Simulation Description:** To demonstration of *Distributed Produce*, I implemented a simulation of two users interacting over a messaging service. -* **Resources** - * [cadCAD Documentation](https://github.com/BlockScience/cadCAD/tree/master/documentation) - * [cadCAD Tutorials](https://github.com/BlockScience/cadCAD/tree/master/tutorials) +* **cadCAD Resources:** + * [Documentation](https://github.com/BlockScience/cadCAD/tree/master/documentation) + * [Tutorials](https://github.com/BlockScience/cadCAD/tree/master/tutorials) * **Terminology:** * ***[Initial Conditions](https://github.com/BlockScience/cadCAD/tree/master/documentation#state-variables)*** - State Variables and their initial values (Start event of Simulation) ```python @@ -133,9 +133,13 @@ cluster_name = 'distibuted_produce' launch_cluster(cluster_name, region, ec2_attributes, bootstrap_actions, instance_groups, configurations) ``` -### 4. Execute Benchmark(s): +### 4. Execute Benchmark(s) on EMR: * **Step 1:** ssh unto master node -* **Step 2:** Spark Submit +```bash +zip -rq distroduce/dist/distroduce.zip distroduce/ +``` +* **Step 2:** ssh unto master node +* **Step 3:** Spark Submit ``` spark-submit --master yarn --py-files distroduce.zip messaging_sim.py `hostname | xargs` ``` diff --git a/distroduce/dist/distroduce.zip b/distroduce/dist/distroduce.zip index 3a0a0acba316d788b30196f04774d08ce90545f0..76c03ea6648f66c0963e45636ea467b45d7e826e 100644 GIT binary patch delta 3237 zcmY+HXH-+m+r^U;2p~o2B^2pZx*{UIN=KTMfD}b)=pBP7ApPngAWftw0#XBt5vfvx zAiWAANYrTouJ=7(o;APOvu9?_`LLgyY@jt8pfc8jk(_}Lzp2NMsCeK_{%J^o zFwmHX5frC~1KMeiOa$QqVgDCaWo!Y!88vVg`mY8ohRFi|E1RF}73;a=mm(E=VaP(O3b47m;(MJtb4P^wy=*ht(dd0s=0=?O99~RJ@ zq4_V}384cu86~02U?5}qUns(4eJaJqFd?9SQ3JC!lo< z1`XWu&{)Y*deu=08}ut~{I{MAzs8Rnj#cXqg{ag|1okPG8mm9O-{#sUB>*bn z&sx1d5MXE2v=}x4yKgEeI1%H zNE9d3QxY$0vFLsEX+^M}g%W3AU%^JOMiswi>2$xDP3cIcN478f<5pIF%!L5eM_pcX0p^{mdT372e(_eVos37Gxpdn# z15tAk5lW2JGd+$PqdCLRb5CmwjmXkON)r}!XWCiKQdV9qi#80oCTp)augW4b;kt6W z)z15pf$-(~(M!}GuViZVAs>JW5@aIM_$?V>qJ7C@;Mzsn$adQLdTW8I&r~a3UP0tt z85{vBc-l(IaP%#OVLMEu-e-y`IS$9-`0kQVrL-YlTp$h@?;RgMC^yfRoLwSQ zFSgiduNb=oTZ$vDz5L0JKuF4P$?n}e{X>n-(D}5@=IdlnPL(k#ZLUl z1T;jpletm_Qh1Fu8P+bpW=CyfWnVmK66w3g5?-4+4q214xzt2Wo+DD4=gXEdlj(ki zIhJ}sf2M=h@y&2M)r9gz*K(_fvp#R{?Vp#-Wo^lH6pKgXhR1A>e(C1xWf3$fq+bD0 zTGPBdPPty_YQBcSS~^?(^vXWF*gH-t)CIk1>{{kDDZ{;jc`gs1^TavaKs0PV10O`( z?r6II2$u7017{XXseKK+{7D(P@F7k8oLizn~JY57Ya zzhv&BRkf9n4jTeRTn2@72S5&MPlfBM*zul7@~T(?I)q2+WC%z)r`sL-efoBAY<)tw z=zdW1bC=@fcKqBCW7|&PdwkY|0{lro>Sku2&#JALS`qm|b=L^x!du(Ref|!-`7I98 z3GB$4p=UGV5%BvO?0h9r^Q{jsv{&-Hqx8N7vr7r4n~!WU(lt3xe!V^aqRl7*IAZZ% zQ@X$X8m;&!(5>_qTe@t{{N$o%&&jro(7|CclAuu;`0!|!U}No(ctOTF=s+&wdfP;N zd(Lc1M5z6FE4i)`X6LWoimYKA&_h}|>QT6>Vzi@Pk({A&Sb>uoN?5D5e_1n|>o9;k zR`;7N4T5sZ@B2gDVnFB)#FI2?kCFX(L#2gZIJ&s3_)J}B>(G;Vmt)_l5CU6qzO^oJ zM1#YLc{(75}-;}{PlyZ*g^q(Eg(9TuJpr+(PjMnw$9JS5z>->uJwdYRwcJc zqqExho5q|htDNKP#ak%5)Ca>z;g=1P>Y@zfWp5s$WlIAjUd#c3-uhFeg)QxQlsu#A zP3W6B_K6cT^oAG|KB;R0!*-3O_VK4T+ZC#Mq308$shwswniG=+jH2p~LjJLY$nXBD zgncN3K?d31kx8T^+Z&}MZ{!seWQ5aLum9SP{*f`?@~juN*w;zJ)4R+nHs@_xkr z!o!i5ka)I=m4q_r7L*6;mUhxne65KEw5jf$g*9Zl`{BDvdzuVyyAe@}rUO zw+F7;N@{0LRwuJ(J-^Lpoz?HisNH#DnrZLZ9H<)-woqzNtKZ{WQ4GY&d{Yvqun9!B zl#-SCDI=5M9>ei|9^7S8mPO_!-XtHFKDv14ipILe?jp3AkQ(yo5+nGIl%7^ZJEruUpb=?zcuFZ7<@;uIl-PnnGqKKaMWkHM!={(J1URxcI8g4eC53=`c zN)9g!AO9-ET?vn2&wVu-Z5JdLX)roAq}7eY4!;0uK)?b#2)I3nOVNCn7KD!7#y8t679{JlX;U7Zx|wIRAk5KyRO zF)%Nb$1 z;yq4w&fb@qtZoV4Fh*$Q?x~WLoA9^4(%!`hCImH0+0a`LuYZOZ`c9Uf=*J1Rd9lIO zc1T{e@8$G3sJ0bS&o16IrTQnazFZU(_Ce*7s?0z)>pB_uhrv4QF>Nxq$VS$zUxbCW zgu3~q`@7fVVrV8~3|}O5mdxTL&6A^Nm7TAq3Nh{2TzR={#yg2NOgUShnONgn-*!}Y z&+^7E0-*`988N0T%^e%CNkx^6GnrmHo=BnP`Z)FT+<5Ns;pbb5z3PV=9UgM-aykXi zxh>Vc&5$)ZhYLULHxAy{JaeH_v>3WMb5ArQB=ndw*oDQ~X-s5$Fig9krFWD{3BmJ= zl$hm-xs4ox2dA91VXNF}pvOfPtkSL1z`n|jmAP}8DSsz*kk|bNF)K#6yTkq>Qj`%9 zS)eFkz|>>V$ybsX48bZt_R}^1tns}FJM9CALoB%;f`$73B}`B}1bUs6k#&)exP@EKDzMe%r;! z9Ds(w04W#*+!LK?Ek9TaYUef~_1} z3ZaoDh-$Ud(kK&1x_N(gMQwD~K%16w^MpBrEn2DryftY*Ao5K0saH#wGascU5FVe_ zqd5?4Ta6spe(TBuVWiFJzbb~OvIF02MJPD{uiYLsmDm0!9i|}K;B<5tFK;9x2!w)z zKu1CRAP?WLi=jat;qIQYHb#1e7Dh4w9(J~jAlhBzZ5LU}dkBv@45Fd?Mgs!T9i_aU zPiF(drkbF81Fm%iefiQ@!^u-Wgzn5<3e+g|5Km4(WSoj)h>5`Y??*>+ zy~`_mV!kGed98j!3g|dLM?P1FmdU08Jqt0P>(Hs&e?>ck+|NV@yCfX_GOq?7-i4Ip za6eayk$qC6I$*PoD1vUOXH;Sg;USCd!d`F8zYgJ|o*$P?PQG+>m|?#ya**w-?h6sv zl}a)xZMJwE2w#(UKelEZdChdNX%x#LViYkxD*4gHy zlhRC>oqL;FMwc^_H-CP7|CHQ4+1pnFnIpp_Tj8-k#eq9J9Lpje;!=wndVV;sb0PLA z9wKgNp@odt;@4K|{ByqaThe8+a^W*7z|H`X7sVs@nA@9vwUkMfFTGmo zxul7(f|_Rp&p-)|VGMQWf{Ud_wW#^faeb@tYE8vFx9Rbj3_&zu;mBra94fg@rNDm) z!@0uL7`MREqh>4CFrgrz_l}gQod6>ItiN2wuM^Lg`Qi9B$-1+d>qq&;r7IEO0lefO z`;y(JdP@um#r^J4e_vyOSA{f^0C;$TJNHC+cKe9w=H1CweR6ZkEc2g2K(j%z~*8SLIrF)&oWgui!uha^wnOYbMD58V;EKjQwqcl#sE z_%n1vwZ=&QYt1;0cSbo~3djwu}!W?6=!hS>m z>y?tADxSm+o3|#ZBc54q(lZVQ>PdS(knDmsNgY$TZ+1--)Lb#vUOd)bG}gXqqZ8+! zXq@eTk^j<92H%jw5bW;Y@-LqJ1i7BhDW+?-DKF*~L%o`M5K^eJeYE$>Rv{Hv-PS_q z66j|b^!(y1Zad?y}(zQ7@kB=8mrp5{N-M{3@Sw^U?Oag=}WbUkI#c zIVw9m7_p)-`iW7616H8~>1er?wD+l2p10@%E{#{~*wSM98=k}y%YA$|U?YnyGSSUo z^PC{(*|KTd{b?@U8`fKJ_ESL`=lm-x&>XpmD~~5FzbN!pKCE3HJQ;+*io3YZYjd9YC1OMksqwZ7j2^X0cuC1$|O(J?QvRC?YbRL zbO#3sJ}ou0;e>8~W9314K&^bIxS4eX4ySk~_l)ADAwD1ppX{WX$$BOHM9u;_SAKi+ zn9Qz8Kz6>rVd`ZLQ9LFYU0nTn=~U7Wq;a5KLXK>yKAQ{+>)P;;O)HqIy za~8t;-+62pWi7diU5Y(kIe#o7`JRSuS4r2~(j~D<*-Ej>?N|Wd!DKjXV79QUi6pam z>$XH45#w5Qf>?6;f_lKE-Jx3BMf(teA~VM1&BT$58xN6v!WwA~QRg8BMu*y(&dqMb zj#h0auD?6=!LsTNroe_=;Ci%Bpa){+C~g4?VZ8NVeO`IbvSlHylbqifrZm;8mEfkv zJ>l&Q#s*xvxR~Y27O+H8QL#{Y9;kGmg&}6OQ(`IO#JlRw#Itlrfs| z@wKJY1balM>tZ(RG1Kd>w^P+{7xVH~^);PcI!E;BvN<92S{)TmdJ!~dHyxW7OGkP& z6Dv;+?5KFn)-6~BdiC4*)`_Yb5U#5aMNRP-UuQY{R6f6B!PxJ?I{(!xvW;FZZ^>Y- zz3xQ3%y8EaoX~VDmU{=<4eu$|@MQ+7dIK0^ww0rQzuNd^| zl<3!zYXv^5Yo<~a1eIHwIiZu9&sE$61JZV({I2~$t}HP}gWN{O$qd8!PurZ=T1Yqp zi$XCcQL{=Jc}(N7?O<1pt}NlRZ1~<_wyI8oT{HYpRLWl7qbz-;n^&yRRd6>EIO=xl z*o@$iFBCK8=`L^ICjD{8v4vYUzG6+c6}I}~BV&;>*X$=59j2DTfnBYY7v&fFBy!xe z9j2gLdq+@q$W6qGu7mG}RhD~x%~v7cdoCiz*ZOkq`o>i}9F~k!V{^5s!=$Po3R)4? zJ%x=lYf}z$wJK`Ze$}{4FYqFrWE)@MRzIol9yElxc+%#_Qds_Z?*Y3#YR^rlo@PF@qn)#~l}fA=H{!s)TbZ|F&Pk7ZAi01JdH66fp;)A> zB6EcEip~HZvGLrytXf~aAN2~C;D@I+N5otdH3AQ7d{!66cl+O8WnxI$4`W!wXQSxbXQ+)k=6RuQZq^G|1{Qh7;>A8DbAr_dIbo{Sh!o<~9c`vuKUWG60`Ly)( zIO-4s;S(0YW;BXk@!P%}O@#P;<)KVXZy(acuYA5Ny>_LU72C^4oMSydceWw6B-+bQ zyoKxlA%=Va)KiT*-61`hX{q+C&%vO0Fx68h%TxZ2RJzpPpBnx{p9nflrLP1XrUf@d z11X_$U~y_xfbtChCPS5}L1>s7H_KlGBVeB<1KveabX>rC1U=B0#Kv|4#C`CmpiGCL zve;n{0Gh-H6ecqOwT2MjQ<@QY-5Z$4!T^(WB)G-zAo?)f63pZOkIM{*VjaO1ffSJd SP=M73|Dd`kdb?0++5Z8Cv)MKP