Skip to content

Software Development News: .NET, Java, PHP, Ruby, Agile, Databases, SOA, JavaScript, Open Source

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Google Open Source Blog
Syndicate content
News about Google's Open Source projects and programs.
Updated: 2 hours 55 min ago

Google Code-in 2016: even more young developers

Thu, 02/23/2017 - 20:04
Google Code-in (GCI), our contest introducing 13-17 year olds to open source software development, wrapped up last month with our largest contest to date: 1,340 students from 62 countries completed an impressive 6,379 tasks! Working with 17 open source organizations, students wrote code, created and edited documentation, designed UI elements and logos, conducted research, developed screencasts and videos teaching others about open source software, and helped find (and fix!) hundreds of bugs.
General statistics
  • 56.4% of students completed three or more tasks (earning themselves a fun Google Code-in 2016 t-shirt)
  • 21% of students were female
  • 30% of the participants from the USA were female
  • This was the first Google Code-in for 1,143 students (85.3%)
Student age2017-02-23_07-48-36.png

Participating schoolsStudents from 550 schools competed in this year’s contest. While Google Code-in is a program for individuals, every year some schools emerge as hot spots of participation. This year, these five schools had the most students taking part:

School NameCountryNumber of ParticipantsDunman High SchoolSingapore185Sacred Heart Convent Senior Secondary SchoolIndia29Jayshree Periwal International SchoolIndia26Colegiul National Aurel VlaicuRomania23Ly Tu Trong Specialized High SchoolsVietnam14
CountriesWe are pleased to have a new country participating in GCI this year: Mauritius! The chart below displays the ten countries with the most students completing at least 1 task.




In June we will welcome all 34 grand prize winners (along with a mentor from each participating organization) for a fun-filled trip to the Bay Area. The trip will include meeting with Google engineers to hear about new and exciting projects, tours of the Google campuses and a fun day exploring San Francisco.

Keep an eye on the Google Open Source Blog in coming weeks for more stats on Google Code-in 2016, plus posts from the mentoring organizations describing some of their experiences with the contests and the work done by “their” students.

We are thrilled that Google Code-in was so popular this year. We hope to continue to grow and expand this contest in the future to introduce even more teenagers to the world of open source software development.

By Stephanie Taylor, Google Code-in Program Manager
Categories: Open Source

Announcing TensorFlow 1.0

Thu, 02/16/2017 - 18:34
Originally posted on the Google Developer Blog

In just its first year, TensorFlow has helped researchers, engineers, artists, students, and many others make progress with everything from language translation to early detection of skin cancer and preventing blindness in diabetics. We're excited to see people using TensorFlow in over 6000 open source repositories online.

Today, as part of the first annual TensorFlow Developer Summit, hosted in Mountain View and livestreamed around the world, we're announcing TensorFlow 1.0:

It's faster: TensorFlow 1.0 is incredibly fast! XLA lays the groundwork for even more performance improvements in the future, and tensorflow.org now includes tips & tricksfor tuning your models to achieve maximum speed. We'll soon publish updated implementations of several popular models to show how to take full advantage of TensorFlow 1.0 - including a 7.3x speedup on 8 GPUs for Inception v3 and 58x speedup for distributed Inception v3 training on 64 GPUs!

It's more flexible: TensorFlow 1.0 introduces a high-level API for TensorFlow, with tf.layers, tf.metrics, and tf.losses modules. We've also announced the inclusion of a new tf.keras module that provides full compatibility with Keras, another popular high-level neural networks library.

It's more production-ready than ever: TensorFlow 1.0 promises Python API stability (details here), making it easier to pick up new features without worrying about breaking your existing code.

Other highlights from TensorFlow 1.0:
  • Python APIs have been changed to resemble NumPy more closely. For this and other backwards-incompatible changes made to support API stability going forward, please use our handy migration guide and conversion script.
  • Experimental APIs for Javaand Go
  • Higher-level API modules tf.layers, tf.metrics, and tf.losses - brought over from tf.contrib.learnafter incorporating skflowand TF Slim
  • Experimental release of XLA, a domain-specific compiler for TensorFlow graphs, that targets CPUs and GPUs. XLA is rapidly evolving - expect to see more progress in upcoming releases.
  • Introduction of the TensorFlow Debugger (tfdbg), a command-line interface and API for debugging live TensorFlow programs.
  • New Android demos for object detection and localization, and camera-based image stylization.
  • Installation improvements: Python 3 docker images have been added, and TensorFlow's pip packages are now PyPI compliant. This means TensorFlow can now be installed with a simple invocation of pip install tensorflow.
We're thrilled to see the pace of development in the TensorFlow community around the world. To hear more about TensorFlow 1.0 and how it's being used, you can watch the TensorFlow Developer Summit talks on YouTube, covering recent updates from higher-level APIs to TensorFlow on mobile to our new XLA compiler, as well as the exciting ways that TensorFlow is being used:

Click here for a link to the livestream and video playlist
(individual talks will be posted online later in the day).The TensorFlow ecosystem continues to grow with new techniques like Foldfor dynamic batching and tools like the Embedding Projector along with updatesto our existing tools like TensorFlow Serving. We're incredibly grateful to the community of contributors, educators, and researchers who have made advances in deep learning available to everyone. We look forward to working with you on forums like GitHub issues, Stack Overflow, @TensorFlow, the discuss@tensorflow.orggroup, and at future events.
By Amy McDonald Sandjideh, Technical Program Manager, TensorFlow

Categories: Open Source

Open-sourcing Google Earth Enterprise

Fri, 02/10/2017 - 21:09
(originally posted on the Geo Developers blog)

We are excited to announce that we are open-sourcing Google Earth Enterprise (GEE), the enterprise product that allows developers to build and host their own private maps and 3D globes. With this release, GEE Fusion, GEE Server, and GEE Portable Server source code (all 470,000+ lines!) will be published on GitHub under the Apache2 license in March.
Screen Shot 2017-01-26 at 2.51.24 PM.pngOriginally launched in 2006, Google Earth Enterprise provides customers the ability to build and host private, on-premise versions of Google Earth and Google Maps. In March 2015, we announced the deprecation of the product and the end of all sales. To provide ample time for customers to transition, we have provided a two year maintenance period ending on March 22, 2017. During this maintenance period, product updates have been regularly shipped and technical support has been available to licensed customers.

Feedback is important to us and we’ve heard from our customers that GEE remains in-use in mission-critical applications. Many customers have not transitioned to other technologies. Open-sourcing GEE allows our customer community to continue to improve and evolve the project in perpetuity. Note that the implementations for Google Earth Enterprise Client, Google Maps JavaScript® API V3 and Google Earth API will not be open sourced. The Enterprise Client will continue to be made available and updated. However, since GEE Fusion and GEE Server are being open-sourced, the imagery and terrain quadtree implementations used in these products will allow third-party developers to build viewers that can consume GEE Server Databases.

We’re thankful for the help of our GEE partners in preparing the codebase to be migrated to GitHub. It’s a lot of work and we cannot do it without them. It is our hope that their passion for GEE and GEE customers will serve to lead the project into its next chapter.

Looking forward, GEE customers can use Google Cloud Platform (GCP) instead of legacy on-premises enterprise servers to run their GEE instances. For many customers, GCP provides a scalable and affordable infrastructure as a service where they can securely run GEE. Other GEE customers will be able to continue to operate the software in disconnected environments. However, we believe that the advantages of incorporating even some of the workloads on GCP will become apparent (such as processing large imagery or terrain assets on GCP that can be downloaded and brought to internal networks, or standing up user-facing Portable Globe Factories).

Moreover, GCP is increasingly used as a source for geospatial data. Google’s Earth Engine has made available over a petabyte of raster datasets which are readily accessible and available to the public on Google Cloud Storage. Additionally, Google uses Cloud Storage to provide data to customers who purchase Google Imagery today. Having access to massive amounts of geospatial data, on the same platform as your flexible compute and storage, makes generating high quality Google Earth Enterprise Databases and Portables easier and faster than ever.

We will be sharing a series of white papers and other technical resources to make it as frictionless as possible to get open source GEE up and running on Google Cloud Platform. We are excited about the possibilities that open-sourcing enables, and we trust this is good news for our community. We will be sharing more information when we launch the code in March on GitHub. For general product information, visit the Google Earth Enterprise Help Center. Review the essential and advanced training for how to use Google Earth Enterprise, or learn more about the benefits of Google Cloud Platform.

Posted by Avnish Bhatnagar, Senior Technical Solutions Engineer, Google Cloud
Categories: Open Source

Google Summer of Code 2016 wrap-up: LabLua

Wed, 02/08/2017 - 18:00
This is the final guest post from the students, mentors and organization administrators that participated in Google Summer of Code (GSoC) 2016. We’ve seen recaps of student work and lessons learned, which you can check out the rest of the series as we gear up for this year’s program.


LabLua is a lab at PUC-Rio dedicated to research on programming languages, with emphasis on the Lua language. Lua is a powerful, fast, lightweight, embeddable scripting language that is used in many industrial applications, and on many embedded systems and games.
We were very happy to participate in Google Summer of Code (GSoC) for the third time, and to mentor eight fine students that all completed their projects successfully. We thank them, and Google, for this extraordinary contribution to our research and development work.
Here is a brief summary of this year's projects:
Next Generation of the LuaRocks test suite - Robert KarasekLuaRocks is the package manager for Lua modules. Its test suite was implemented as a big shell script that performed only black-box testing and ran only on Linux. The goal for this project was to port the test suite to Lua, improving its portability and allowing more types of tests so we could improve test coverage.
Robert ported the test suite to Lua using Busted. His new test suite, now merged into LuaRocks, runs on Linux and Mac OS X, accessible via Travis CI, as well as Windows, accessible via AppVeyor. 
This was a welcome addition, bringing greater confidence to developers. Robert improved the checks in existing tests and wrote many new ones, including a new mock-server for testing a client API for uploading packages to the repository.
Typed Lua Typechecker - Tomasz Dyczek 
Typed Lua provides static type checking for the Lua language. Typed Lua extends the syntax of Lua 5.3 to introduce type annotations, and performs local type inference for more precise detection of unannotated expressions.
Tomasz implemented the core of Typed Lua in Haskell. Tomasz's implementation parses code written in a syntax close to the abstract syntax of Typed Lua, then type checks the generated AST. Besides providing a support for testing and reasoning about new features, Tomasz's typechecker can be also used to validate tests to be included in Typed Lua's test suite.
Classes and Generics for Typed Lua - Kevin Clancy
Kevin worked on the implementation of a class system for Typed Lua. He also added parametric polymorphism (generics) for classes and existing Typed Lua types, such as functions and tables.
Kevin's work currently lives in its own branch, but will be merged into the main branch soon. Meanwhile, Kevin has written a detailed post explaining all the features he implemented
Improving Error Reporting in PEG Parsers - Matthew Allen Go 
LPegLabel is an extension of LPeg, a pattern matching tool for Lua, based on Parsing Expression Grammars (PEGs). LPegLabel supports labeled failures, a facility that improves error reporting and recovery for PEG-based parsers.
The goal of this project was to use LPegLabel to write parsers with good error reporting. These parsers could then be used by the Lua community and also serve as a guide for LPegLabel users. Because LPegLabel is a young tool, another important contribution was to improve the tool's usability.
Matthew achieved both goals. He developed a parser for Lua 5.3, which has been incorporated into the new release of lua-parser (1.0.0), and improved LPegLabel’s usability with work on its API and documentation.
Improving elasticsearch-lua tests and build - Dhaval Kapil
Elasticsearch is a distributed and scalable search engine written in Java that offers a REST API accessed through JSON. During GSoC 2015, Dhaval implemented elasticsearch-lua, a client for the Lua language following a model similar to clients written in Python and PHP.
During GSoC 2016, Dhaval worked on improving elasticsearch-lua. He added a test suite, documented the entire codebase, and updated the current client to work with the newest version of Elasticsearch.
Dhaval went above and beyond, creating a new library called luaver. This work was motivated by having to frequently switch between different versions of Lua while developing the test suite. A full blog post about his project can be found here.
Admin Center and Elasticsearch integration for Sailor - Nikhil Ramesh 
Sailor is a web framework with a model-view-controller (MVC) architecture. Like other web frameworks, such as Ruby on Rails and Django, it is designed to make development faster by making some assumptions and conventions and encouraging principles like Don’t Repeat Yourself (DRY).
Nikhil focused on extending Sailor. The first feature he worked on was an Admin Center, which is a web interface for configuring an application. He also integrated Sailor and elasticsearch-lua, allowing Elasticsearch indexes to be stored as Sailor Models. His work is currently pending as a pull request and will soon be merged.
Extending the online tutorial of Céu with Emscripten and SDL - Margarit Vicentiu
Céu is a language for developing reactive applications such as video games and embedded systems. Its compiler generates output in plain C to integrate easily with the underlying platform (e.g. Arduino, SDL). For this project, we wanted to integrate Céu with Emscripten in order to run applications in a web browser.
Vicentiu started with Céu’s online tutorial, which is a server-side application: the user types code in a text area and hits the send button; the server receives the code, executes it, and sends the output back to the user. During the summer, Vicentiu made most of the examples compile with Emscripten and run in real-time on the user’s screen.
Our next goal is to make the graphical examples with user interactions also work in the browser, and Vicentiu plans to continue contributing to the project to achieve this goal.
An automatic generator of WSDL documents for LuaSOAP - Victor Dias
LuaSOAP is a library for working with the Simple Object Access Protocol (SOAP). WSDL is an XML format for describing network services; it is used to describe operations, messages and types offered by Web Services.
This summer Victor extended LuaSOAP's WSDL support by building a software layer for the automatic generation of WSDL documents. This new layer eases the description of most WSDL "bureaucracy" -- types, operations, ports, messages -- which have no counterparts in Lua. He also improved the test suite and the documentation. Victor's work will be integrated into the next version of LuaSOAP.
By Ana Lúcia de Moura, Organization Administrator for LabLua
Categories: Open Source

Google Summer of Code 2016 wrap-up: CloudCV

Mon, 02/06/2017 - 18:00
This guest post is part of our ongoing series of posts from the students, mentors and organization administrators who participated in Google Summer of Code (GSoC), a program which gets university students contributing to open source software.

Google Summer of Code 2016 was a memorable one for CloudCV. Despite being a relatively “young” organization (this is just our second year as a mentor organization), there were many excellent applicants who put a tremendous amount of effort into their proposals and ramp-up tasks. It was difficult to choose!

CloudCV began in the summer of 2014 as a research project within the Machine Learning and Perception Lab at Virginia Tech, with the ambitious goal of democratizing computer vision and machine learning. We’re run exclusively by students and are working to enable developers, researchers, and fellow students to leverage artificial intelligence technology as a service and to share state of the art algorithms with the research community.

In line with this goal, we decided to build two tools that cater to computer vision researchers and hobbyists alike: CloudCV-fy your code and CloudCV-IDE. Though building two new platforms from the ground up was going to be challenging, our students’ motivation was overwhelming and their performance surpassed all expectations. We even demonstrated their work at CVPR 2016, a top-tier computer vision conference!

CloudCV-fy

A recurring use case for computer vision researchers, and many others, is to build a web-based demo and REST API to demonstrate the capabilities of their creations to the world. But web development involves writing hundred of lines of additional code across multiple languages (HTML, CSS, JavaScript, etc), which takes time away from research.


Our first student, Ashish Chaudhary, took on this problem by building CloudCV-fy. Over many iterations of design and development, Ashish delivered a tool that allows a user to simply write lightweight wrappers around their machine learning model/library and be done. CloudCV-fy automatically builds web-based interactive demos for them -- no need to tinker with HTML, CSS or JavaScript. Code to demo. Done.

The demo can be hosted on our servers, the user’s own server or any third party cloud service. As a result of this, researchers can focus on what they do best: designing and training models. CloudCV handles the rest. You can learn more in the write-up Ashish did on his blog.

CloudCV-IDE

There has been an explosion in the number of deep learning frameworks and it is difficult for researchers to keep up with all the latest tools. CloudCV-IDE, built by student Gaurav Gupta, addresses this by allowing a user to build a deep learning network with a drag-and-drop interface, then export to the deep learning framework of their choice (Caffe, TensorFlow, etc).

Gaurav also added support to import model configuration files in order to visualize any architecture. This is one of the first attempts to do this.



By the end of the summer, Gaurav delivered a great UI to visualize models with robust support for Caffe and TensorFlow back-ends. This was a successful start that we plan to build on by supporting more frameworks and facilitating collaborative building of deep learning models.

Overall, this was a highly productive GSoC for CloudCV. Our tools are under active development and we welcome contributions and ideas for new features.

We will definitely apply for GSoC 2017. If you are a student interested in participating we encourage you to get involved early! Feel free to reach out to us on our Gitter channel or on our mailing list.

By Viraj Prabhu, Organization Administrator for CloudCV
Categories: Open Source

Googlers on the road: FOSDEM 2017

Wed, 02/01/2017 - 19:22
The new year is off to an excellent start as we wrap up the 7th year of Google Code-in, ramp up for the 13th year of Google Summer of Code, and return from connecting with our compatriots in the open source community down under at Linux.conf.au. Next up? We’re headed to FOSDEM, Europe’s famed non-commercial and volunteer-organized open source conference.

FOSDEM_logo.pngFOSDEM logo licensed under CC BY.
FOSDEM is hosted in Brussels on the Université libre de Bruxelles campus and runs the weekend of February 4-5. It’s a unique event in the spirit of the free and open source software and is free to the public. This year they are expecting 8,000+ attendees.

We’re looking forward to talking face-to-face with some of the thousands of former students, mentors and organization administrators who have participated in our student programs. A few of them will even be giving talks about their recent Google Summer of Code experience.

If you’d like to say hello or chat about our programs, you’ll be sure to find a Googler or two at our table. You’ll also find a number of Googlers in the program schedule:

Saturday, February 4th

2:00pm    Bazel: How to build at Google scale by Klaus Aehlig
3:25pm    Copyleft in Commerce: How GPLv3 keeps Samba relevant in the marketplace by Jeremy Allison

Sunday, February 5th

10:40am  gRPC 101: Building fast and efficient microservices by Ray Tsang
10:50am  Is the GPL a copyright license or a contract under U.S. law? by Max Sills
12:45pm  The state of Go: What to expect in Go 1.8 by Francesc Campoy
1:00pm    Analyze terabytes of OS code with one query by Felipe Hoffa (more info)
2:50pm    Like the ants: Turn individuals into a large contributing community by Dan Franc

See you there!

By Josh Simmons, Open Source Programs Office

Categories: Open Source

Introducing Draco: compression for 3D graphics

Mon, 01/30/2017 - 23:00
3D graphics are a fundamental part of many applications, including gaming, design and data visualization. As graphics processors and creation tools continue to improve, larger and more complex 3D models will become commonplace and help fuel new applications in immersive virtual reality (VR) and augmented reality (AR).  Because of this increased model complexity, storage and bandwidth requirements are forced to keep pace with the explosion of 3D data.

The Chrome Media team has created Draco, an open source compression library to improve the storage and transmission of 3D graphics. Draco can be used to compress meshes and point-cloud data. It also supports compressing points, connectivity information, texture coordinates, color information, normals and any other generic attributes associated with geometry.

With Draco, applications using 3D graphics can be significantly smaller without compromising visual fidelity. For users this means apps can now be downloaded faster, 3D graphics in the browser can load quicker, and VR and AR scenes can now be transmitted with a fraction of the bandwidth, rendered quickly and look fantastic.


Sample Draco compression ratios and encode/decode performance*
Transmitting 3D graphics for web-based applications is significantly faster using Draco’s JavaScript decoder, which can be tied to a 3D web viewer. The following video shows how efficient transmitting and decoding 3D objects in the browser can be - even over poor network connections.


Public domain Discobolus model from SMK National Gallery of Denmark.

Video and audio compression have shaped the internet over the past 10 years with streaming video and music on demand. With the emergence of VR and AR, on the web and on mobile (and the increasing proliferation of sensors like LIDAR) we will soon be swimming in a sea of geometric data. Compression technologies, like Draco, will play a critical role in ensuring these experiences are fast and accessible to anyone with an internet connection. More exciting developments are in store for Draco, including support for creating multiple levels of detail from a single model to further improve the speed of loading meshes.

We look forward to seeing what people do with Draco now that it's open source. Check out the code on GitHub and let us know what you think. Also available is a JavaScript decoder with examples on how to incorporate Draco into the three.js 3D viewer.

By Jamieson Brettle and Frank Galligan, Chrome Media Team

* Specifications: Tests ran with textures and positions quantized at 14-bit precision, normal vectors at 7-bit precision. Ran on a single-core of a 2013 MacBook Pro.  JavaScript decoded using Chrome 54 on Mac OS X.
Categories: Open Source

Google Summer of Code 2016 wrap-up: GitHub

Mon, 01/30/2017 - 21:45
Every year open source organizations, mentors, and university students come together to build and improve open source software through Google Summer of Code (GSoC). This guest post is part of a series of blog posts from people who participated in GSoC 2016.

GitHub-Mark-120px-plus.png

Open source maintainers at GitHub mentored 5 students in Google Summer of Code last year. The students did great work that we’d like to highlight and congratulate them on:
Updates to GitHub Classroom GitHub Classroom helps teachers automate their work and interact with students in issues and pull requests. Last summer two students took on projects to help teachers work more efficiently and with greater insight into their classrooms.
Classroom Project #1 Cheng-Yu Hsu is a student who worked to implement new features suggested by teachers using GitHub Classroom, including due dates for assignment submissions and visualizations of classroom activities. In reflecting on the project, Cheng-Yu said:

"Having a great community is one of the most important factors of a successful open source project, so participating [in] the community is also a huge part of this project. It is great to have chances responding to user feedback, helping people resolve issues and brainstorming new features with them."
Classroom Project #2Shawn Ding worked on student identifiers and team management for GitHub Classroom. This means that teachers using GitHub Classroom can use things such as student emails to identify their assignments. Teachers can also now manage their students and teams of students using GitHub Classroom via drag and drop in the settings page which then updates the data on GitHub.
Front-end controls for JekyllJekyll Admin is a Jekyll plugin that provides users with a traditional CMS-like graphical interface to author content and administer Jekyll sites from the comfort of their browser. GSoC student Mert Kahyaoğlu has been using Facebook’s React framework to create the front-end that will allow you to write a new post, edit existing pages or add new files. And it will all work with GitHub Pages.

Best of all, Mert's plugin allows people to author content and administer Jekyll sites without knowledge of command line or installing an external text editor like Atom. Once installed, Jekyll users start their site as they would normally and simply append “/admin” to their site's URL to launch the WordPress-like administrative interface. Jekyll Admin's initial release is ready for use on your own site.
Octokit.net Alexander Efremov added support to Octokit.net for interacting with the GitHub API using a repository ID, alongside the existing support for providing the owner and repository name. This means integrators do not have to update their systems when a repository changes ownership. The changes to support these APIs were rolled out incrementally over a number of pull requests, and 0.21 release of Octokit.net made these new APIs available to the public.

We had a great time mentoring these students on their projects last year!

By Carol Smith, John Britton and Brandon Keepers, Organization Administrators for GitHub

Categories: Open Source

Google Summer of Code 2016 wrap-up: Orange

Mon, 01/30/2017 - 21:36
This post is part of our ongoing series of guest posts from the students, mentors and organization administrators who participated in Google Summer of Code (GSoC). GSoC is a program that pairs university students with mentors for a summer where they apply their computer science skills to building open source software.


Orange Data Mining is a data mining suite with visual programming and interactive data analysis at its core. Orange was developed at Bioinformatics Lab at University of Ljubljana, Slovenia, it is written mainly in Python, and you can find it hosted on GitHub.

This was our third Google Summer of Code and we were given five slots and decided to select students based on two criteria: their proposal joined with their coding skills and the importance of the project to our organization.

Great work was done over the summer and we are proud to present our students’ projects!

Recommender Systems add-on by Salva Carrion

Salva independently implemented a new Orange3 add-on for recommender systems. He developed a scripting library for collaborative filtering for the core of the add-on, which includes a number of published matrix factorization algorithms. The scripting library is then further extended to include GUI-based widgets for visual programming.

Educational add-on by Primož Godec

Primož took on a task of developing a series of educational widgets for Orange3. The end result was a full Orange3-Educational add-on with four widgets that can be used to demonstrate key data mining and machine learning procedures in the classroom. These widgets are useful for helping beginners understand the inner workings of key algorithms in data mining, and for teachers to be able to visually explain the various methods. They include interactive and step-by-step visualizations of k-means, polynomial classification, and gradient descent.



Text add-on by Aliaxey Sukharevich

Orange3-Text add-on was already an active project before GSoC, but Aliaxey took it to another level. Twitter and Wikipedia public RESTful services were introduced as widgets to allow acquisition of data from new sources. Many widgets were boosted with new functionalities and methods (e.g. HDP, LDA and LSP methods in Topic Modelling widget). Preprocessing was redesigned and reimplemented such that it now handles n-grams and POS Tagging.



CN2 Rule Induction by Matevž Kren

The goal of this project was to implement a CN2 rule induction algorithm, and Orange widgets for learning and exploration of inferred classification rules. At the heart of the project is an implementation of a scripting library, which can be easily extended with additional divide and conquer algorithms or its components.

Porting Orange codebase to Pandas by Sašo Stanovnik

This was a gargantuan task and Sašo handled it beautifully. The goal was to consolidate Orange data structures and management routines to support data from Pandas. Sašo redesigned Orange data management core, did a massive amount of refactoring and improvements and removed legacy and unused code. The biggest challenge was of course preserving as much compatibility with the existing Orange interaction as possible while providing full Pandas flexibility. The result is a functional Pandas-based core Orange.

All contributions were committed on GitHub (Orange3, Orange3-Text, Orange3-Recommendation and Orange3-Educational repositories) and most of them are already pip-installable. The only contribution that has not yet been merged is the migration to Pandas, which will require adaptation and careful compatibility checking of other components of the system.

We are extremely grateful to be given the chance to participate in Google Summer of Code and to have had such amazing students at our lab. We can’t wait to apply again next year!

By Ajda Pretnar, Organization Administrator for Orange
Categories: Open Source

Announcing the Google Code-in 2016 Winners!

Mon, 01/30/2017 - 18:58
Drum roll please! We are very proud to announce the 2016 Google Code-in (GCI) Grand Prize Winners and Finalists. Each year we see the number of student participants increase, and 2016 was no exception: 1,340 students from 62 countries completed an impressive 6,418 tasks. Winners and Finalists were chosen by the 17 open source organizations and are listed alphabetically below.
First is a list of our Grand Prize winners. These 34 teens completed an astounding 842 total tasks. Each Grand Prize winner will be flown to the Google campus for four days this summer to meet with Google engineers and enjoy the Bay Area.

GRAND PRIZE WINNERS Name Organization Country Matthew Marting Apertium United States Shardul Chiplunkar Apertium United States Michal Hanus BRL-CAD Czech Republic Sudhanshu Agarwal BRL-CAD India Alexandru Bratosin CCExtractor Development Romania Evgeny Shulgin CCExtractor Development Russian Federation Joshua Pan Copyleft Games Group United States Shriank Kanaparti Copyleft Games Group India Dhanat Satta-awalo Drupal Thailand Utkarsh Dixit Drupal India Kaisar Arkhan FOSSASIA Indonesia Oana Roşca FOSSASIA Romania Raefaldhi Amartya Junior Haiku Indonesia Vanisha Kesswani Haiku India Ilya Bizyaev KDE Russian Federation Sergey Popov KDE Russian Federation Anshuman Agarwal MetaBrainz India Daniel Hsing MetaBrainz Hong Kong Dhruv Shrivastava Mifos India Sawan Kumar Mifos India Ong Jia Wei, Isaac Moving Blocks Singapore Scott Moses Sunarto Moving Blocks Indonesia Mira Yang OpenMRS United States Nji Collins OpenMRS Cameroon Cristian García Sugar Labs Uruguay Tymon Radzik Sugar Labs Poland August van de Ven SCoRe Netherlands Deniz Karakay SCoRe Turkey Jacqueline Bronger Systers Germany Soham Sen Systers India Filip Grzywok Wikimedia Poland Justin Du Wikimedia United States Sampriti Panda Zulip India Tommy Ip Zulip United Kingdom
And below are the Finalists. Each of these 51 students will receive a digital certificate of completion, a GCI t-shirt and hooded sweatshirt.

FINALISTS Name Organization Bror Hultberg Apertium Kamil Bujel Apertium Ngadou Sylvestre Apertium Apratim Ranjan Chakrabarty BRL-CAD Tianyue Gao BRL-CAD Trung Nguyen Hoang BRL-CAD Danila Fedorin CCExtractor Development Manveer Basra CCExtractor Development Matej Plavevski CCExtractor Development Daniel Wee Soong Lim Copyleft Games Group Jonathan Pan Copyleft Games Group Oscar Belletti Copyleft Games Group Ashmith Kifah Sheik Meeran Drupal Heervesh Lallbahadur Drupal Neeraj Pandey Drupal Adarsh Kumar FOSSASIA Ridhwanul Haque FOSSASIA Sanchit Mishra FOSSASIA Dmytro Shynkevych Haiku Stephanie Fu Haiku Tudor Nazarie Haiku Harpreet Singh KDE Sangeetha S KDE Spencer Brown KDE Daniel Theis MetaBrainz Divya Prakash Mittal MetaBrainz Tigran Kostandyan MetaBrainz Illia Andrieiev Mifos Justin Du Mifos Tan Gemicioglu Mifos J Young Kim Moving Blocks Maxim Borsch Moving Blocks Quinn Roberts Moving Blocks Shivani Thaker OpenMRS Tenzin Zomkyi OpenMRS Yusuf Karim OpenMRS Emily Ong Hui Qi Sugar Labs Euan Ong Sugar Labs Pablo Salomón Ortega Quintana Sugar Labs Basil Najjar SCoRe Jupinder Parmar SCoRe Thuận Nguyễn SCoRe Muaaz Kasker Systers Muhammed Shamil K Systers Phoebe Fletcher Systers David Siedtmann Wikimedia Nikita Volobuev Wikimedia Yurii Shnitkovskyi Wikimedia Cynthia Lin Zulip Rafid Aslam Zulip Robert Hönig Zulip

The Google Open Source Programs Office is proud to run this contest each year. The quality of work from our participating students is incredible, and each year we look forward to meeting our Grand Prize winners in person. It’s exciting to see the next generation of coders emerge! We also owe a huge debt of gratitude to all of the mentors who helped guide each of the participants through their tasks. Without their tireless work over the past 7 weeks, GCI would not be possible.

Stay tuned to the open source blog - we’ll regularly post Google Code-in 2016 stories in the upcoming months including a full breakdown of contest statistics, wrap-up posts from the organizations, student highlights and more.

By Mary Radomile, Open Source Programs Office
Categories: Open Source

Join the POSSE Workshop on Student Involvement in Humanitarian Free and Open Source Software

Thu, 01/26/2017 - 18:00
Are you a university or college instructor interested in providing students with experience in real-world projects? Are you interested in supporting participation in humanitarian free and open source software (HFOSS)? If so, join the Professor's Open Source Software Experience (POSSE) workshop being held at Google’s San Francisco Office, April 20-22, 2017.

Over 100 faculty members have attended past workshops and there is a growing community of faculty members helping students learn within HFOSS projects. This three-stage faculty workshop will prepare you to support student participation in open source projects. In the workshop, you will:

  • Become part of the community of educators which involves students in HFOSS
  • Learn how to support student learning within real-world project environments
  • Motivate students and raise their appreciation of computing for social good
  • Meet and collaborate with instructors who have similar interests and goals

Workshop Format

Stage 1: Starts February 23, 2017 with online activities. These activities will take 2-3 hours per week and include interaction among workshop instructors and participants.

Stage 2: The face-to-face workshop will be held at the Google San Francisco office, April 20-22, 2017. Participants include the workshop organizers, POSSE alumni and members of the open source community.

Stage 3: Comprises online activities and interactions among small groups. Participants will have support while involving students in an HFOSS project in the classroom.

Please click here to learn more about the POSSE workshop in April.

How to Apply

To apply, please complete and submit the application by February 13th. Prior work with FOSS projects is not required. The POSSE workshop committee will send you a confirmation email to notify you of the status of your application by February 23rd, 2017.

Participant Support

POSSE is supported by the National Science Foundation (NSF) and Google. NSF funding will provide two nights lodging and meals during the workshop. Travel costs will be covered up to $500. At this time,we can only support US-based faculty members. However, if you can support your own travel, please do submit an application!

Why is Google participating?

Google is participating in order to help educators overcome challenges identified in the POSSE workshop held last June, and to better support FOSS education in academia. We are very happy to host the first POSSE workshop located on the west coast of the United States.

See you in San Francisco this April!

By Helen Hu, Open Source Programs Office
Categories: Open Source

Google Summer of Code 2016 wrap-up: QEMU

Mon, 01/23/2017 - 18:00

This post is part of a series of guest posts from students, mentors and organization administrators reflecting on Google Summer of Code (GSoC) 2016.




QEMU is a machine emulator and virtualizer. It can run operating systems and programs made for one machine (e.g. an ARM board) on a different machine (e.g. your own PC).  It is also used by hypervisors like KVM and Xen to virtualize x86, ARM, POWER and other architectures so virtual machines can run at near-native performance.

This year was QEMU's sixth time participating in Google Summer of Code and we had 6 students from around the world. Here are highlights from GSoC 2016:

Pranith Kumar worked on Multi-threaded TCG, a larger effort to make QEMU's just-in-time compiler multi-core friendly. He studied the memory ordering CPU instructions for many architectures supported by QEMU and devised a TCG opcode that can represent the memory ordering semantics. His patches allow TCG to correctly translate memory ordering instructions (previously they were ignored!). This is one piece of the puzzle to support multi-threaded translation and will help emulation scale to many cores. Pranith also contributed to testing and debugging multi-threaded TCG issues that he encountered while contributing to this cutting-edge part of QEMU.

Gerard Garcia worked on packet capture support for AF_VSOCK in Linux, tcpdump and Wireshark. QEMU recently gained support for the virtio-vsock device that allows socket communication between guest and host. The AF_VSOCK address family didn't support packet capture yet, making it hard to debug and observe communication between applications. Gerard’s patches implement the kernel interfaces and tcpdump/Wireshark functionality needed to monitor traffic over AF_VSOCK. This project was challenging because it required getting code merged in multiple open source projects. Gerard pulled it off with great skill!

Every year QEMU applies to participate in GSoC and we’re delighted for the opportunity to mentor students interested in systems programming, emulation, compilers and low-level code like firmware. You are welcome to get in touch on IRC or our mailing lists any time of the year to find out about contributing to QEMU. We also have a wiki page with small tasks you can get started on as your first step in QEMU programming.

By Stefan Hajnoczi, Organization Administrator for QEMU

Categories: Open Source

Now accepting organization applications for Google Summer of Code 2017

Thu, 01/19/2017 - 22:15


We’re heading into the 13th year of Google Summer of Code (GSoC) and are now accepting applications for open source organizations. GSoC is a global program that gets student developers involved as open source contributors. Students spend three months working under the guidance of mentors on projects to expand and improve open source software.

Last year we had 178 open source organizations and 1,200 students participate. Open source organizations include open source projects and the umbrella organizations that often serve as their fiscal sponsors.



Do you represent a free or open source software organization? Are you seeking new contributors? (Of course!) Do you love the challenge and reward of mentoring new developers in your community? Apply to be a mentor organization for Google Summer of Code! Starting today we are accepting applications from open source projects who would like to serve as mentor organizations for enthusiastic student developers.

The deadline to apply is February 9 at 16:00 UTC. Organizations chosen for GSoC 2017 will be posted on February 27.

Please visit the program site for more information on how to apply, a detailed timeline of important deadlines and general program information. We also encourage you to check out the Mentor Manual and join the discussion group. You can also learn more by exploring our series of guest posts written by mentor organizations who participated in GSoC 2016.

Best of luck to all of the applicants!

By Josh Simmons, Open Source Programs Office
Categories: Open Source

Google Code-in 2016: another record breaking year

Mon, 01/16/2017 - 22:18
Today we celebrate the closing of the 7th annual Google Code-in (GCI) which, like last year, was bigger and better than ever. Mentors from each of the 17 organizations are busy reviewing the last of the work submitted by student participants.

Each organization will pick two Grand Prize Winners who will receive a trip to Google’s Northern California headquarters this summer where they will meet Google engineers, see exciting demos and presentations and enjoy a day of adventure in San Francisco. You can learn about the experiences of the 2015 Grand Prize Winners in our short series of wrap-up blog posts. We’ll announce the new Grand Prize Winners and the Finalists here on January 30.

We would like to congratulate all of the new and returning students who participated this year. We’re thrilled with the turnout: over the last seven weeks, 1,374* students from 62 countries completed 6,397* tasks in the contest.

And a HUGE thanks to the people who are the heart of our program: the mentors and organization administrators. These volunteers spend countless hours creating and reviewing hundreds of tasks. They teach the young students who participate in GCI about the many facets of open source development, from community standards and communicating across time zones to version control and testing. We couldn’t run this program without you!

By Josh Simmons, Open Source Programs Office

* These numbers will increase over the coming days as mentors review the final work submitted by students.
Categories: Open Source

JanusGraph connects the past and future of Titan

Thu, 01/12/2017 - 19:04
We are thrilled to collaborate with a group of individuals and companies, including Expero, GRAKN.AI, Hortonworks and IBM, in launching a new project — JanusGraph — under The Linux Foundation to advance the state-of-the-art in distributed graph computation.



JanusGraph is a fork of the popular open source project Titan, originally released in 2012 by Aurelius, and subsequently acquired by DataStax. Titan has been widely adopted for large-scale distributed graph computation and many users have contributed to its ongoing development, which has slowed down as of late: there have been no Titan releases since the 1.0 release in September 2015, and the repository has seen no updates since June 2016.

This new project will reinvigorate development of the distributed graph system to add new functionality, improve performance and scalability, and maintain a variety of storage backends.

The name "Janus" comes from the name of a Roman god who looks simultaneously into the past to the Titans (divine beings from Greek mythology) as well as into the future.

All are welcome to participate in the JanusGraph project, whether by contributing features or bug fixes, filing feature requests and bugs, improving the documentation or helping shape the product roadmap through feature requests and use cases.

Get involved by taking a look at our website and browse the code on GitHub.

We look forward to hearing from you!

By Misha Brukman, Google Cloud Platform
Categories: Open Source

Apache Beam graduates to a top-level project

Tue, 01/10/2017 - 18:09
Please join me in extending a hearty digital “Huzzah!” to the Apache Beam community: as announced today, Apache Beam is an official graduate of the Apache Incubator and is now a full-fledged, top-level Apache project. This achievement is a direct reflection of the hard work the community has invested in transforming Beam into an open, professional and community-driven project.

11 months ago, Google and a number of partners donated a giant pile of code to the Apache Software Foundation, thus forming the incubating Beam project. The bulk of this code composed the Google Cloud Dataflow SDK: the libraries that developers used to write streaming and batch pipelines that ran on any supported execution engine. At the time, the main supported engine was Google’s Cloud Dataflow service with support for Apache Spark and Apache Flink in development); as of today there are five officially supported runners. Though there were many motivations behind the creation of Apache Beam, the one at the heart of everything was a desire to build an open and thriving community and ecosystem around this powerful model for data processing that so many of us at Google spent years refining. But taking a project with over a decade of engineering momentum behind it from within a single company and opening it to the world is no small feat. That’s why I feel today’s announcement is so meaningful.

With that context in mind, let’s look at some statistics squirreled away in the graduation maturity model assessment:

  • Out of the ~22 large modules in the codebase, at least 10 modules have been developed from scratch by the community, with little to no contribution from Google.
  • Since September, no single organization has had more than ~50% of the unique contributors per month.
  • The majority of new committers added during incubation came from outside Google.

And for good measure, here’s a quote from the Vice President of the Apache Incubator, lifted from the public Apache incubator general discussions list where Beam’s graduation was first proposed:

“In my day job as well as part of my work at Apache, I have been very impressed at the way that Google really understands how to work with open source communities like Apache. The Apache Beam project is a great example of this and is a great example of how to build a community." -- Ted Dunning, Vice President of Apache Incubator

The point I’m trying to make here is this: while Google’s commitment to Apache Beam remains as strong as it always has been, everyone involved (both within Google and without) has done an excellent job of building an open source project that’s truly open in the best sense of the word.
This is what makes open source software amazing: people coming together to build great, practical systems for everyone to use because the work is exciting, useful and relevant. This is the core reason I was so excited about us creating Apache Beam in the first place, the reason I’m proud to have played some small part in that journey, and the reason I’m so grateful for all the work the community has invested in making the project a reality.
Naturally, graduation is only one milestone in the lifetime of the project, and we have many more ahead of us, but becoming top-level project is an indication that Apache Beam now has a development community that is ready for prime time.
That means we’re ready to continue pushing forward the state of the art in stream and batch processing. We’re ready to bring the promise of portability to programmatic data processing, much in the way SQL has done so for declarative data analysis. We’re ready to build the things that never would have gotten built had this project stayed confined within the walls of Google. And last but perhaps not least, we’re ready to recoup the vast quantities of text space previously consumed by the mandatory “(incubating)” moniker accompanying all of our initial mentions of Apache Beam!
But seriously, whatever your motivation, please consider joining us along the way. We have an exciting road ahead.
By Tyler Akidau, Apache Beam PMC and Staff Software Engineer at Google
Categories: Open Source

Google Summer of Code 2016 blog post round-up

Tue, 01/10/2017 - 07:26
We’re publishing guest posts from Google Summer of Code (GSoC) students, mentors and organizations every week and more are coming. Many have already written GSoC wrap-up posts on their own blogs, so we’ve rounded them up for you to explore.


Static types in Python, oh my(py)!” by Tim Abbott, org admin for Zulip
“We posted mypy annotations as one of our project ideas for Google Summer of Code (GSoC). We found an incredible student, Eklavya Sharma, for the project. Eklavya did the vast majority of the hard work of annotating Zulip. Amazingly, he also found the time during the summer to migrate Zulip to use virtualenvs and then upgrade Zulip to Python 3!”


A road from Google Summer of Code student to organization administrator” by Araz Abishov, org admin for HISP
“Google has created unprecedented opportunity both for young developers and open source communities, which I think everyone should take advantage of. GSoC is more than just a three months internship, and I hope that this post will be a good example of how it can change anyone’s life.”


Summer of Code 2016: Wrapping it up” by Martin Braun, org admin for GNU Radio
“This summer was a great summer in terms of student participation. All three students will be presenting their work (either in person, or via poster) at this year’s GNU Radio Conference in Boulder, Colorado.”


2016 Google Summer of Code Wrap-Up” by Ed Cable, org admin for Mifos Initiative
“Each year GSoC continues to unite and grow our community in different ways. Once again, we received incredibly valuable contributions to our Mifos X web and mobile clients this summer; most importantly we have cultivated numerous passionate contributors that will be a part of our community long into the future.”


Road to GSoC 2016” by Minh Chu, student who worked on Neverland for KDE
“I was nervous about choosing a project. So many projects and requirements! After many hours, I finally decided to write a proposal for KDE’s Neverland Theme Builder and was accepted.”


Git Rev News” by Christian Couder, mentor for Git
“Such performance improvements as well as the code consolidations around the sequencer are of course very nice. It is interesting and satisfying to see that they are the result of building on top of previous work over the years by GSoC students, mentors and reviewers.”

Elasticsearch Lua II" by Dhaval Kapil, student who worked with LabLua
“My GSoC project this year was entitled ‘Improve elasticsearch-lua tests and builds’ and was a continuation of the work that I had done last year. Apart from adding a test suite for elasticsearch-lua and making it robust, I also decided to work on the documentation of the code."

Google Summer of Code 2016 Conclusion” by Amine Khaldi, org admin for ReactOS
“Students stumble upon many of the same difficulties ReactOS' own senior developers encountered during their early days, including that ever painful but necessary step to using a proper debugger instead of relying on printf statements in the code.”


My Journey in Open Source / How to Get Started Contributing” by Nelson Liu, student who worked on scikit-learn for PSF
“The best way to get started is to simply jump in! There are a myriad of ways to contribute to an open source project. Obviously, writing code to fix bugs, add new features, or enhance existing ones are useful. However, you don't have to write code to help out!”

Google Summer of Code 2016 Student Projects” by Pankaj Nathani, org admin for BuildmLearn
“Many open source projects like ours really benefit from this initiative of Google. Not only do we get large number of university students interested to work on our projects during summer; we also gain new long term contributors and project maintainers."

Lasp and the Google Summer of Code” by Borja o’Cook, student who worked on Lasp for BEAM Community
“All in all, it's been an amazing experience. I've received a lot of support from my mentors and teammates; the Lasp team is full of incredible people.”


GSoC 2016 Students in TEAMMATES” by Damith C. Rajapakse, org admin for TEAMMATES
“We had our biggest batch of students (7 students) in GSoC 2016, selected from 93 proposals, and representing 4 countries and 4 universities, working on TEAMMATES (an online feedback management system for education) and related sub projects.”


User-friendly encryption now in Drupal 8!” by Colan Schwartz, mentor for Drupal
“There were several students interested in the topic, and wrote proposals to match. Talha Paracha's excellent proposal was accepted, and he began in earnest. With Adam Bergstein (nerdstein) and I mentoring him, Talha successfully worked through all phases of the project.”


GSoC with Shogun” by Sanuj Sharma, student who worked on Shogun
“This was an excellent learning experience for me and I got to work with people from different countries (UK, Russia, Singapore, Germany) and cultures. I highly recommend students to participate in Google Summer of Code by looking for projects that interest them because having open source experience is highly beneficial, especially for programmers.”


We have wrap-up posts coming out every week so stay tuned for more. If you’re interested in participating in Google Summer of Code 2017, you can find details here.

By Josh Simmons, Open Source Programs Office
Categories: Open Source

Open source down under: Linux.conf.au 2017

Sun, 01/08/2017 - 22:53
It’s a new year and open source enthusiasts from around the globe are preparing to gather at the edge of the world for Linux.conf.au 2017. Among those preparing are Googlers, including some of us from the Open Source Programs Office.

This year Linux.conf.au is returning to Hobart, the riverside capital of Tasmania, home of Australia’s famous Tasmanian devils, running five days between January 16 and 20.
Circle_DevilTuz.pngTuz, a Tasmanian devil sporting a penguin beak, is the Linux.conf.au mascot.
(Artwork by Tania Walker licensed under CC BY-SA.)The conference, which began in 1999 and is community organized, is well equipped to explore the theme, "the Future of Open Source," which is reflected in the program schedule and miniconfs.

You’ll find Googlers speaking (listed below) as well as participating in the hallway track. Don’t miss our Birds of a Feather session if you’re a student, educator, project maintainer, or otherwise interested in talking about outreach and student programs like Google Summer of Code and Google Code-in.

Monday, January 16th
12:20pm The Sound of Silencing by Julien Goodwin
1:20pm   An Open Programming Environment Inspired by Programming Games by Josh Deprez

Tuesday, January 17th
All day    Community Leadership Summit X at LCA

Wednesday, January 18th
2:15pm   Community Building Beyond the Black Stump by Josh Simmons

Thursday, January 19th
4:35pm   Using Python for creating hardware to record FOSS conferences! by Tim Ansell

Friday, January 20th
1:20pm   Linux meets Kubernetes by Vishnu Kannan

Not able to make it to the conference? Keynotes and sessions will be livestreamed, and you can always find the session recordings online after the event.

We’ll see you there!

By Josh Simmons, Open Source Programs Office
Categories: Open Source

Google Summer of Code 2016 wrap-up: Oppia

Fri, 01/06/2017 - 18:00
Google Summer of Code (GSoC) is an annual program that encourages university students to become open source contributors. This guest post is part of a series of blog posts from the open source projects and organizations that participated in GSoC 2016.

The Oppia project makes it easy for anyone to create lightweight, interactive online lessons that simulate personal tutoring. These activities, called “explorations,” can be shared with others around the world as standalone tutorials (such as Programming with Carla and Quadratic Equations), or embedded in websites to supplement an existing course (such as “Take Your Medicine” on edX and Computational Thinking for Educators).

2016 was Oppia’s first year participating in GSoC and it was a blast! More students flocked to our ideas page than we had expected, and our Gitter channel was full of people saying hello and looking for starter projects. Over the course of the summer, with the help of two capable and enthusiastic students, we were able to bring the following new features to the Oppia codebase:

A new creator dashboard -- Avijit Gupta


An important principle of Oppia is that lessons can be easily improved over time -- it’s hard to figure out all the possible ways a student can go wrong at the outset, but it’s much easier to respond appropriately to a new misconception that arises.

Each creator on Oppia has a “creator dashboard” which allows them to see the lessons they’ve created, as well as the feedback they’ve received from learners. Avijit completed a full revamp of this page, updating its design (for both desktop and mobile) and finding ways to display all the necessary information in an intuitive way so that creators can easily improve their lessons while getting feedback on their teaching.

The new creator dashboard.
In addition, Avijit added functionality allowing creators to view student misconceptions that were not well-addressed, to make it easier for them to improve the feedback for those answers. He has continued to help out with the Oppia open source project as a maintainer and reviewer, even after GSoC, and is mentoring other contributors who are working on further improvements to the creator dashboard. You can read more about the project in his GSoC writeup!

Speed improvements -- Vishal Gupta


In order to improve the accessibility of lessons for students with poor internet connectivity, Vishal’s project aimed to make Oppia speedier and less bandwidth-intensive. He started by implementing a performance testing framework to benchmark his efforts, and also integrated it with our continuous integration system in order to protect against performance regressions. He then turned his efforts to caching as many static resources as possible, implementing a cache slug system that causes new files to be downloaded only after a new release is made.

In addition, Vishal removed JavaScript code that was inlined in the main templates, and refactored it out into an external script which could then be cached for better performance. You can read more about this project in his post on the Oppia blog.

We’d like to extend our grateful thanks not only to Avijit and Vishal, but also to our many willing and enthusiastic mentors, and to Google for supporting our open source work with GSoC.

Join us in helping improve educational opportunities for students around the world. If you’d like to subscribe to news and updates about Oppia’s participation in GSoC, you can sign up to the oppia-gsoc-announce mailing list -- or, if you’re already feeling enthusiastic, you can start helping out with the project right away!

By Ben Henning and Sean Lip, Organization Administrators for Oppia
Categories: Open Source

Grumpy: Go running Python!

Wed, 01/04/2017 - 18:00
Google runs millions of lines of Python code. The front-end server that drives youtube.com and YouTube’s APIs is primarily written in Python, and it serves millions of requests per second! YouTube’s front-end runs on CPython 2.7, so we’ve put a ton of work into improving the runtime and adapting our application to work optimally within it. These efforts have borne a lot of fruit over the years, but we always run up against the same issue: it's very difficult to make concurrent workloads perform well on CPython.

To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.
MeatGrinder.png
So we asked ourselves a crazy question: What if we were to implement an alternative runtime optimized for real-time serving? Once we started going down the rabbit hole, Go seemed like an obvious choice of platform since its operational characteristics align well with our use case (e.g. lightweight threads). We wanted first class language interoperability and Go’s powerful runtime type reflection system made this straightforward. Python in Go felt very natural, and so Grumpy was born.

Grumpy is an experimental Python runtime for Go. It translates Python code into Go programs, and those transpiled programs run seamlessly within the Go runtime. We needed to support a large existing Python codebase, so it was important to have a high degree of compatibility with CPython (quirks and all). The goal is for Grumpy to be a drop-in replacement runtime for any pure-Python project.

Two design choices we made had big consequences. First, we decided to forgo support for C extension modules. This means that Grumpy cannot leverage the wealth of existing Python C extensions but it gave us a lot of flexibility to design an API and object representation that scales for parallel workloads. In particular, Grumpy has no global interpreter lock, and it leverages Go’s garbage collection for object lifetime management instead of counting references. We think Grumpy has the potential to scale more gracefully than CPython for many real world workloads. Results from Grumpy’s synthetic Fibonacci benchmark demonstrate some of this potential:



Second, Grumpy is not an interpreter. Grumpy programs are compiled and linked just like any other Go program. The downside is less development and deployment flexibility, but it offers several advantages. For one, it creates optimization opportunities at compile time via static program analysis. But the biggest advantage is that interoperability with Go code becomes very powerful and straightforward: Grumpy programs can import Go packages just like Python modules! For example, the Python snippet below uses Go’s standard net/http package to start a simple server:

from __go__.net.http import ListenAndServe, RedirectHandler

handler = RedirectHandler('http://github.com/google/grumpy', 303)
ListenAndServe('127.0.0.1:8080', handler)

We’re excited about the prospects for Grumpy. Although it’s still alpha software, most of the language constructs and many core built-in types work like you’d expect. There are still holes to fill — many built-in types are missing methods and attributes, built-in functions are absent and the standard library is virtually empty. If you find things that you wish were working, file an issue so we know what to prioritize. Or better yet, submit a pull request.

Stay Grumpy!

By Dylan Trotter, YouTube Engineering
Categories: Open Source