Many modern enterprise applications rely on search to some extent. As of Nov 2016 the most popular search engine is Elasticsearch. It is an open source engine based on Apache Lucene. The need to perform search arose in my home project as well. I chose Elasticsearch for the engine and readily dived into the tutorials. My methodology for writing interactions with the 3rd party systems is to create Facade APIs within Test Driven Development process. The tests for indexing and retrieving documents worked flawlessly, but the test results for the search queries got me puzzled. I have formal training in search engines within Coursera Data Mining specialization, thus I know concepts like TF-IDF. The hope was to get the relevance scores and match them precisely to the numbers computed by formulas in the tutorials.

Basic index for 4 test documents returned me the numbers vastly different from my expectations… After some googling I turned on the “explain” functionality and was up to an even bigger shock: the returned scores didn’t match the scores in the explain section. I started suspecting the unthinkable: the relevance calculations are broken! Elasticsearch tutorial confirmed my worst fears… well, it rather explained to me how little I know about the real search engines. After couple more hours of comparing numbers the discrepancies were decomposed into an optimization feature, a bug pretending to be a feature, and a bug. The optimization feature is that several shards are created for each index and documents are randomly distributed between those shards. The relevance calculations are only performed within each shard for DEFAULT search type. Setting search type to DFS_QUERY_THEN_FETCH forces shard statistics to be combined into a single IDF calculation, thus leading to values closer to the expected numbers. However, the “explain” functionality always employs the DEFAULT search type leading to a mismatch, hence a bug. A bug pretending to be a feature is in really coarse-grained rounding of the relevance norm. The discrepancies reach 15%, which hurts testing.

Here is my review for Big Data Capstone Project course offered on Coursera in Jul 2016. The course represents the final project for the Big Data specialization, it does not have separate rankings, while I passed with 98.2% score.
Technologies/Material: As a final project, the course does not have lectures, but rather brief descriptions of relevant project parts each week. The project is about making suggestions on how to increase revenue of a company promoting a fictional game “Catch the Pink Flamingo”. A lot of simulated game data is made available to the learners. The part assigned each week represents a separate area of big data analytics: data exploration, classification, clustering, and graph analysis. The suggested technologies are: Splunk, KNIME, Apache Spark, and Neo4j, respectively. As usual within the specialization instead of free exploration a “correct” path is given along with substantial help on the way. The assignment each week is peer graded with the ability to submit multiple times and get regraded. Grading asks to compare learners’ numbers with the correct numbers, which means that almost everyone gets correct answers on their second attempt. Unfortunately, many people slack off on their first attempt or simply submit an empty report. At the end of the course a final report with a powerpoint presentation are submitted and also peer graded.
Instructor/lectures: the task instructions are given by Amarnath Gupta and Ilkay Altintas. The course offers a realistic view of a job of a Data Scientist: analyze all available data to increase revenue of a company, improve retention rates, suggest the ways of development, and, most importantly, make presentations to the management. The instructors emphasize each week that the company’s bottom line is of the utmost importance. Even though the specialization is called Big Data, there is no emphasize on especially large volumes of data or on distributed computations, thus we are in the Data Science realm.

Here is my review for Graph Analytics for Big Data course offered on Coursera in Feb 2016. The course is ranked 2.5 out of 5, while I passed with 99.4% score.
Technologies/Material: The course provides introduction to graph theory with practical examples of graph analytics. Most of examples and homework is done in Neo4j, a leading graph database. The last assignment employs GraphX API in Spark. Since graph databases are so different from regular databases, the special graph query language called Cypher was developed to write code for Neo4j. Extensive Cypher tutorial and executable code samples grouped by topics are given. Graph analytics offers simple answers to many questions. The discussed graph techniques are Path Analytics, Dijkstra algorithm and its variations, Connectivity Analytics, Community Analytics, and Centrality Analytics.
Instructor/lectures: the course is taught by Amarnath Gupta, an Associate Director of San Diego Supercomputer Center. Amarnath is an amazing instructor. The course is well taught with just the right speed and the right amount of material given. In my view, he succeeded in making an introduction to graphs, while not oversimplifying the concepts.

Here is my review for Machine Learning with Big Data course offered on Coursera in Jan 2016. The course got 2.0 out of 5 rankings and I passed it with 100% score.
Technologies/Material: The course provides basic theory and some exercises on popular machine learning techniques after presenting business justification and ML pipeline. The presented techniques are decision trees, association rules, and clustering. Exercises are largely done in KNIME with some parts in Apache Spark. Thankfully, the course has copyable code samples and provides basic information on how to get started with KNIME. The assignments require digging into non-trivial details of KNIME from its documentation/Internet/forums. For me the course provided valuable insights and examples of decision trees and association rules, which not many other courses offer.
Instructor/lectures: The course is taught by Natasha Balac, who provides most of business background, and Paul Rodriguez, who is a technical person. The presentation is organized better than in previous courses, though the depth of the material is often not sufficient for solid learning. Some slides can be reused to present Big Data to managers.

The autonomous driving technology is still early in its development, but more and more self-driving cars appear on the road. To facilitate progress many manufacturers put some elements out as driver assist features: adaptive cruise control, collision warning, emergency braking, lane keeping aid, park assist, road sign recognition, pedestrian and cyclist recognition.
I risked getting one of those vehicles with many driver assist features and try it out by myself. Here are the impressions from the first week in 2016 Volvo S60 Platinum.

1. Extremely relaxing to not push gas or brake pedals even in stop&go traffic!
2. Can readily go 5mph above the preset speed on steep declines – bummer.
3. A person can see several cars ahead and predict, if the car right ahead will suddenly decelerate – cruise control isn’t that smart yet.
4. Cruise control’s braking power is insufficient in ~3% of cases for the shortest following distance.
5. Luckily, collision warning kicks in very early on, if the person ahead suddenly brakes.
6. Lane keeping aid applies short and efficient bursts of torque – funny to find my hands in a slightly different position after 0.5 second.
7. Following through turns works just fine meaning adaptive cruise control performs well in the city.
8. No, car doesn’t accelerate uncontrollably into turns even at high preset speed and nobody around – thanks Volvo engineers for resolving this safety concern.
9. Picking stationary vehicles and red lights are the only weak points, but yes, those are hard to resolve.
10. Car always complains about me coming too close to the curb, while parking – such a fussy baby.
11. Rear cross-traffic alert is a must-have addition to the rear view camera, if one doesn’t really want to turn their head for backing.
12. BLIS indicator at the base of the rearview mirror is infinitely better than on the mirror itself.
13. No, I don’t really want to test emergency braking for pedestrians and byciclists and hopefully it won’t ever need to engage…
14. Not sure if the Speed Limit Pilot in new Mercedes is useful – my car sometimes reads the signs wrong + naturally old speed limit will display till it sees another sign.

No, my car is not fully autonomous, but here is the Volvo vision http://www.volvocars.com/intl/about/our-innovation-brands/intellisafe/intellisafe-autopilot/this-is-autopilot.

For many years I performed most of my data manipulation in Wolfram Mathematica.
With broad support for functional programming, it was nice and simple to apply a function on any arbitraty vector or matrix. The resultant code was really nice and short. A “for” loop was very rarely needed. Having switched to Java 6 and 7, I find such manipulations with data substantially less pleasant. Given the standard packages of Java 7, a “for” loop is unavoidable and the code becomes substantially longer.
Google Guava partially solves the problem with method like “Lists.transform“.
However, in this case the function still has to be explicitly defined.
Let me illustrate the transition on the example of parsing search results in Youtube API. Java 7 code

List<String> videoIds = new ArrayList<>();
for(SearchResult searchResult: searchListResponse.getItems())
          videoIds.add(searchResult.getId().getVideoId());

Java 7 code with functional programming in Guava:

Function<SearchResult, String> func = new Function<SearchResult, String>() {
            @Override
            public String apply(SearchResult searchResult) {
                  return searchResult.getId().getVideoId();
            }
     };
List<String> videoIds Lists.transform(searchListResponse.getItems(), func);

The code became longer and less transparent rather than becoming simpler. This is a known caveat in working with Google Guava under Java 6 and 7.
Luckily, Java 8 brings such needed simplications making the above code truly a one-liner.
Lambda expressions save the day

List<String> videoIds = 
   Lists.transform(searchListResponse.getItems(), d -> d.getId().getVideoId());

A simple method can instead be passed as a method reference

List<ResourceId> resourceIds = 
        Lists.transform(searchListResponse.getItems(), SearchResult::getId);

The third level of Java certification is becoming an Oracle Certified Java Expert. Unlike for the previous 2 levels, the learners have a choice of passing either “Enterprise JavaBeans Developer“, or “Web Component Developer“, or “Web Services Developer” exam. I chose to concentrate on the Web Services exam as the one covering most in-demand material, which also aligns well with my job at Deutsche Bank. I was spending about 1hr every day at work to work through the tutorials and code samples + I’m exposed to Web Services within my work projects. It still took about 1 year to prepare, as the exam is quite hard.

Here is what I did to prepare:
Step 1. Made a study plan based on my preparations for the professional certification. I wanted to read an official Oracle tutorial, a book on Web Services, some online study guides, look through and run some sample code, also check all use cases of Web Services within my applications at work.

Step 2. Read Java EE 7 Oracle tutorial. Only about 25% of topics are relevant for the exam, but I wanted to get a feel for all enterprise technology, which comes out-of-the box with Java. Some constructs, e.g. listeners and interceptors, are widely used in software development. Some non-trivial concepts of Enterprise Java Beans (EJB) do find their way into Web Services exam.

Step 3. Bought and worked through “Java Web Services: Up and Running” book by Martin Kalin. The book describes the development of Web Services from XML RPC to state transfer and method invocations on proxies. A variety of examples provide hands-on experience with Web Servers (Tomcat), build tools (Ant), generation and usage of WSDL files & XSD schemas. I was able to transition most code from GlassFish to Tomcat and successfully run it via IntelliJ IDEA.

Step 4. Read through Oracle Certified Expert in Web Services guide by Mikalai Zaikin. This one is specifically aimed at the exam. It has a good coverage of Jersey client and asynchronous JAX-WS method invocations.

Step 5. Skimmed through SCDJWS 5 exam study guide by Ivan A. Kriszan, which can be found here. The guide is for the previous version of the exam, but it still very informative. It covers in detail XML and WSDL as languages and schemas + Web Services definitions as special instances of objects written in those languages. This guide was the first to describe business registry (UDDI) as well as XML parsing in great detail. It has the whole chapters dedicated to security, interoperability, and design principles.

Step 6. Read through the specifications of JAX-RS 1.19 and JAX-WS 2.2 used on the exam. The specifications themselves are very dry and can only be used to clarify unclear concepts, e.g. how precisely a RESTful server chooses, how to parse the URI and which method to invoke with which input and output content types. JAX-WS specification does a great job elaborating on precise rules for SOAP/Logical handler invocations in case of chain interruptions/exceptions.

Step 7. Read through the APIs and implementations of relevant classes and packages from URLConnection to WebServiceRef: Jersey Core, Jersey Server, Jersey Client, JAX-WS.

Step 8. In parallel to steps 6 and 7 did Enthuware practice exams. I was lucky to be able to take 5 exams on consecutive Saturdays and work through the answers. The exams served to create some mnemonic rules on, e.g. which SEI and SIB methods are exposed; what the names are of Service, Port, PortType, Operation etc in relation to names of classes/methods and customizations. The exams did motivate me to study vague topics such as patterns for authentication and authorization + WS-I basic profile. Mock tests revealed the great variety of non-trivial high-level design questions. They helped me master Jersey Client and RESTful services questions, which aren’t all that hard.

Step 9. Developed a strategy for the exam. Split 55 questions into 5 groups of 11 and split 90 minutes allotted time into 18min intervals, so that I can be in full control of time. Mastered the elimination technique and technique of remembering and reusing answers to similar questions. The real exam was harder than the mock tests and some design questions were quite ambiguous. Fortunately, I gained enough knowledge and practical experience over the course of a year to get a passing score of 72% from the first try!

In sum, it was a tough year, but I do now feel like a Java Expert. This for some time concludes my largely theoretical studies of programming in favor of more practical applications of my knowledge.

Here is my review of Hadoop Platform and Application Framework course offered on Coursera in Nov 2015. Course has the worst rankings of 1 out of 5 (very bad), while I passed with a 100% grade.

Technologies/Material: The course goes in more detail into Big Data databases + SQL/HQL languages to operate on data such as Pig and Hive. Splunk appears as a disjoint tool, whose primary purpose is to provide ease-of-use to data analysts & business analysts. It is obligatory to sign-up for Splunk e-mails and you will get those e-mails. In contrast, introduction to Spark Dataframes by Andrea Zonca in Week 5 is very well done and can serve as a starting point for writing own Big Data workflows. In addition, some real Pig and Hive scripts are provided for reference. Homework assignments are more interesting and more numerous, than for the previous course. They already hint at real world applications. At the time the course was offered, the Big Data specialization was downgraded in difficulty from Intermediate to beginner. As an improvement over the previous course, the lecture slides are finally provided. Quiz questions are edited without any notifications to students(!), for which the course rightly deserves its ranking. I posted hints in a course forum on how some quiz questions should be modified in order for the answers marked by the grader as correct to actually be correct. Hopefully, this made the experience of students slightly more bearable.

Instructor/lectures: the course has a large number of instructors (3), which does not help with coherent elaboration of the material. For this course, even the material in the same week is sometimes delivered by 2 people. All instructors are affiliated with the University of California, San Diego. Lectures often turn into reading the tutorials and even leaving an example half-way through without achieving a positive result(!) – see HBase tutorial in Week 1. Lectures end with 11min of black screen (!). Following very negative student feedback for this course the next course “Machine Learning With Big Data” is postponed.

Here is my review of Hadoop Platform and Application Framework course offered on Coursera in Oct 2015. Course has bad rankings of 2 out of 5 (bad), while I passed with a 99% grade.

Technologies/Material: The course briefly touches on every major Big Data tool present in Cloudera Virtual Machine: MapReduce framework + HDFS constituting the core of Hadoop; YARN, Spark, Pig, Hive, HBase. While some material in the early weeks is not useful and/or repeats Cloudera tutorials, the introduction to Spark by Andrea Zonca in Week 5 is very well done and can serve as a reference guide. Non-native Python interface to Hadoop is chosen for sparse exercises. In turn, presented Python interface to Spark is native and provides great interactive capabilities. There is a substantial number of code examples, which are often shown only in lecture videos without any ability to copy them(!), e.g. for Hive. The course name fluctuated and eventually converged to a longer version to reflect the broader set of topics. Programming assignments are very simplistic and require mostly copy-pasting from examples. At the time the course was offered, the Big Data specialization was assigned Intermediate difficulty.

Instructor/lectures: the course has an unusually large number of instructors (4), which hinders coherent elaboration of the material. There is a substantial overlap between different weeks and substantial differences in focus as well as in presentation style: Natasha Balac is largely discussing management of Big Data zoo, Andrea Zonca provides highly technical lectures, while the other two guys are somewhere in between. All instructors are affiliated with the University of California, San Diego.

Here is my review of Introduction to Big Data course offered on Coursera in Sep-Oct 2015. Course has controversial rankings of 3 out of 5 (poor), while I passed with a 100% grade.

Technologies/Material: the course is highly unusual and experimental in nature. It is aimed to entice and encourage newcomers to Big Data with a collection of factual information. Any meaningful homework is absent. Out of stated ~15hrs, the course required under 3hrs to complete. The effective cost of $18/hr might only be justified, when course is viewed as a prerequisite for completing a Big Data specialization. The specialization has received “intermediate” difficulty ranking, but later was rightfully downgraded to a “beginner” specialization.

Instructor/lectures: Natasha Balac was holding IT manager positions since receiving her PhD from Vanderbilt in 2002. She has business experience rather than technical experience with Big Data, hence the course is oriented on business context.