Skip to content

Software Development News: .NET, Java, PHP, Ruby, Agile, Databases, SOA, JavaScript, Open Source

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Open Source

Open-sourcing DeepMind Lab

Google Open Source Blog - Wed, 12/07/2016 - 18:00
Originally posted on DeepMind Blog

DeepMind's scientific mission is to push the boundaries of AI, developing systems that can learn to solve any complex problem without needing to be taught how. To achieve this, we work from the premise that AI needs to be general. Agents should operate across a wide range of tasks and be able to automatically adapt to changing circumstances. That is, they should not be pre-programmed, but rather, able to learn automatically from their raw inputs and reward signals from the environment. There are two parts to this research program: (1)  designing ever-more intelligent agents capable of more-and-more sophisticated cognitive skills, and (2) building increasingly complex environments where agents can be trained and evaluated.

The development of innovative agents goes hand in hand with the careful design and implementation of rationally selected, flexible and well-maintained environments. To that end, we at DeepMind have invested considerable effort toward building rich simulated environments to serve as  â€ślaboratories” for AI research. Now we are open-sourcing our flagship platform,  DeepMind Lab, so the broader research community can make use of it.

DeepMind Lab is a fully 3D game-like platform tailored for agent-based AI research. It is observed from a first-person viewpoint, through the eyes of the simulated agent. Scenes are rendered with rich science fiction-style visuals. The available actions allow agents to look around and move in 3D. The agent’s “body” is a floating orb. It levitates and moves by activating thrusters opposite its desired direction of movement, and it has a camera that moves around the main sphere as a ball-in-socket joint tracking the rotational look actions. Example tasks include collecting fruit, navigating in mazes, traversing dangerous passages while avoiding falling off cliffs, bouncing through space using launch pads to move between platforms, playing laser tag, and quickly learning and remembering random procedurally generated environments. An illustration of how agents in DeepMind Lab perceive and interact with the world can be seen below:

At each moment in time, agents observe the world as an image, in pixels, rendered from their own first-person perspective. They also may receive a reward (or punishment!) signal. The agent can activate its thrusters to move in 3D and can also rotate its viewpoint along both horizontal and vertical axes.

Artificial general intelligence research in DeepMind Lab emphasizes navigation, memory, 3D vision from a first person viewpoint, motor control, planning, strategy, time, and fully autonomous agents that must learn for themselves what tasks to perform by exploring their environment. All these factors make learning difficult. Each are considered frontier research questions in their own right. Putting them all together in one platform, as we have, represents a significant new challenge for the field.


DeepMind Lab is highly customisable and extendable. New levels can be authored with off-the-shelf editor tools. In addition, DeepMind Lab includes an interface for programmatic level-creation. Levels can be customised with gameplay logic, item pickups, custom observations, level restarts, reward schemes, in-game messages and more. The interface can be used to create levels in which novel map layouts are generated on the fly while an agent trains. These features are useful in, for example, testing how an agent copes with unfamiliar environments. Users will be able to add custom levels to the platform via GitHub. The assets will be hosted on GitHub alongside all the code, maps and level scripts. Our hope is that the community will help us shape and develop the platform going forward.



DeepMind Lab has been used internally at DeepMind for some time (example). We believe it has already had a significant impact on our thinking concerning numerous aspects of intelligence, both natural and artificial. However, our efforts so far have only barely scratched the surface of what is possible in DeepMind Lab. There are opportunities for significant contributions still to be made in a number of mostly still untouched research domains now available through DeepMind Lab, such as navigation, memory and exploration.

As well as facilitating agent evaluation, there are compelling reasons to think that it may be fundamentally easier to develop intelligence in a 3D world, observed from a first-person viewpoint, like DeepMind Lab. After all, the only known examples of general-purpose intelligence in the natural world arose from a combination of evolution, development, and learning, grounded in physics and the sensory apparatus of animals. It is possible that a large fraction of animal and human intelligence is a direct consequence of the richness of our environment, and unlikely to arise without it. Consider the alternative: if you or I had grown up in a world that looked like Space Invaders or Pac-Man, it doesn’t seem likely we would have achieved much general intelligence!

Read the full paper here.

Access DeepMind's GitHub repository here.

By Charlie Beattie, Joel Leibo, Stig Petersen and Shane Legg, DeepMind Team


Categories: Open Source

Schedule Announced for Devoxx US 2017

Eclipse News - Mon, 12/05/2016 - 20:00
Schedule for DevoxxUS 2017 is now available! 200+ talks, Hackergarten, hands-on labs, quickies, BoFs & more!
Categories: Open Source

CFP Closes in 1 Week | Eclipse Converge

Eclipse News - Mon, 12/05/2016 - 18:30
Discover the tracks, and submit your passion to Eclipse Converge 2017!
Categories: Open Source

Why I contribute to Chromium

Google Open Source Blog - Mon, 12/05/2016 - 18:00
This is a guest post by Yoav Weiss who was recently recognized through the Google Open Source Peer Bonus Program for his work on the Chromium project. We invited Yoav to share about his work on our blog.

I was recently recognized by Google for my contributions to Chromium and wanted to write a few words on why I contribute to the project, other rendering engines and the web platform in general. I also wanted to share how it helped me evolve as a developer and why more people should contribute to the web platform for their own benefit.
The web platformI’ve written before about why I think the web platform is an extremely important asset for humanity and why we should make sure it'll thrive for years to come. It enables the distribution of knowledge to the corners of the earth and has fundamentally changed our world. Yet, compared to the amount of users (billions!) and web developers (millions), there are only a few hundred engineers working on maintaining and improving the platform itself.

That means that there are many aspects of the platform that are not as well maintained as they should be. We're at a real risk of a "tragedy of the commons" scenario, where despite usage and utility, the platform will collapse under its own weight because maintaining it is nobody's exclusive problem.
How I got startedPersonally, I had been working on web performance for well over a decade before I decided to get more involved and lend my hand in building the platform. For a large part of my professional life, browsers were black boxes. They were given to us by the browser gods and that's what we had to work with for the next few years. Their undocumented bugs and quirks became gospel, passed from senior engineers to their juniors.

Then at some point, that situation changed. Slowly but surely, open source browsers started picking up market share. No longer black boxes, we can actually see what happens on the inside!

I first got involved by joining the responsive images discussions and the Responsive Images Community Group. Then, I saw a tweet from RICG's chair calling to develop a prototype of the current proposal to prove its feasibility and value. And I jumped in.

I created a prototype using Chromium and WebKit, demoed it to anyone that was interested, worked on the proposals and argued the viability of the proposals' approach on the various mailing lists. Eventually, we were able to get some browser folks on board, improve the proposals and their fit to the rest of the platform, and I started working on an implementation.

The amount of work this required was larger than I expected. Eventually I managed to ship the feature in Blink and Chromium, and complete large parts of the implementation in WebKit as well. WOOT!
Success! Now what?After that project was done, I started looking into what I should do next. I was determined to continue working on browsers and find a gig that would let me do that. So I searched for an employer with a vested interest in the web and in making it faster, who would be happy to let me work on the platform's client - the web browser.

I found such an employer in Akamai, where I have been working as a Principal Architect ever since. As part of my job I'm working on our performance optimization features as well as performance-related browser features, making sure they make it into browsers in a timely fashion.
Why you should contribute, tooNow, chances are that if you're reading this, you're also relying on the web platform for your job in one way or another. Which means that there's a chance that it also makes sense for your organization to contribute to the web platform. Let’s explore the reasons:
1. Make sure work is done on features you care aboutIf you're like me, you love the web platform and the reach it provides you, but you're not necessarily happy with all of it. The web is great, but not perfect. Since browsers and web standards are no longer black boxes, you can help change that.

You can work on standards and browsers to change them to include your use-case. That's immense power at your fingertips: put in the work and the platform evolves for all the billions of users out there.

And you don’t have to wait years before new features can be used in production like with yesteryear's browser changes. With today’s browser update rates and progressive enhancement, you’ll probably be able to use changes in production within a few months.
2. Gain expertise that can help you do your job betterKnowing browser internals better can also give you superpowers in other parts of your job. Whenever questions about browser behavior arrive, you can take a peek into the source code and have concrete answers rather than speculation.

Keeping track of standards discussions give you visibility into new browser APIs that are coming along, so that you can opt to use those rather than settle for sub-optimal alternatives that are currently available.
3. Grow as an engineerWorking on browsers teaches you a lot about how things work under the surface and enables you to understand the internals of modern browsers, which are extremely complex machines. Further, this work allows you to get code reviews from the world's leading experts on these subjects. What better way to grow than to interact with the experts?
4. It's a fun and welcoming communityContributing to the web platform has been a great experience for me. Working with the Chromium project, in particular, is always great fun. The project is Google backed, but there are many external contributors and the majority of work and decisions are being done in the open. The people I've worked with are super friendly and happy to help. All in all, it's really fun!
Join usThe web needs more people working on it, and working on the web platform can be extremely beneficial to you, your career and your business.

If you're interested in getting started with web standards, the Discourse instance of the web Platform Incubator Community Group (or WICG for short) is where it's at (disclaimer: I'm co-chairing that group). For getting started with Chromium development, this is the post for you.

And most important, don't be afraid to ask the community. People on blink-dev and IRC are super friendly and will be happy to point you in the right direction.

So come on over and join the good cause. We'll be happy to have you!

By Yoav Weiss, Chromium contributor
Categories: Open Source

December 2016, “Staff Pick” Project of the Month – Tcl

SourceForge.net: Front page news - Mon, 12/05/2016 - 06:22

For our December “Staff Pick” Project of the Month, we selected Tcl, an interpreted language and very portable interpreter for that language.

Tcl is a very powerful yet easy-to-learn dynamic programming language that’s been widely used since its creation in 1988. It is highly embeddable and extensible, cross-platform and suitable for a very wide range of uses. These include web and desktop applications, networking, administration, testing and many more.

Open source and business-friendly, Tcl is a mature yet continuously evolving language. It is highly-rated among the projects featured on SourceForge, and has been nominated as “Community Choice” Project of the Month in previous months.

Learn more about Tcl by visiting their website.

 

[ Download Tcl ]

Categories: Open Source

Projects of the Week, December 5, 2016

SourceForge.net: Front page news - Mon, 12/05/2016 - 06:05

Here are the featured projects for the week, which appear on the front page of SourceForge.net:

SMPlayer

SMPlayer is a free media player for Windows and Linux with built-in codecs that can also play YouTube videos. One of the most interesting features of SMPlayer: it remembers the settings of all files you play. So you start to watch a movie but you have to leave… don’t worry, when you open that movie again it will be resumed at the same point you left it, and with the same settings: audio track, subtitles, volume… SMPlayer is a graphical user interface (GUI) for the award-winning MPlayer, which is capable of playing almost all known video and audio formats. But apart from providing access for the most common and useful options of MPlayer, SMPlayer adds other interesting features like the possibility to play YouTube videos subtitles. Note: for those people complaining about malware in the windows installer: be sure you download SMPlayer from the official website. Our installer is completely safe and free of malware.
[ Download SMPlayer ]


championify-logo Championify

Championify brings you the critical information you need to succeed in League of Legends by downloading all the latest items from sites like Champion.gg, Lolflavor, and KoreanBuilds and importing them into your game. Get the highest winning builds, most popular skills upgrades and more and achieve the best in League of Legends with Championify.
Championify is available in 39 languages, packed with new features and supports Windows and OSX.
[ Download Championify ]


Password Safe

Password Safe is a password database utility. Users can keep their passwords securely encrypted on their computers. A single Safe Combination unlocks them all.
[ Download Password Safe ]


Manjaro Community Torrents

This project is for download the Manjaro Officials and Community releases using a bittorrent client (console or graphical)
[ Download Manjaro Community Torrents ]


vJoy

This project was originally designed to provide an open-source replacement for PPJoy. The product, at this point, consists of virtual joystick devices that is seen by the system as a standard joystick but its position-data is written to it by a feeder application. An existing feeder application that takes advantage of this product is SmartPropoPlus. If you are an application writer you can very easily write an application that controls a joystick (e.g. mouse-to-joystick, keyboard-to-joystick). If you are a beginner in device drivers you can take this code and enhance it to support more (or less) axes, buttons or POVs.
[ Download vJoy ]


fre:ac – free audio converter

fre:ac is a free audio converter and CD ripper for various formats and encoders. It features MP3, MP4/M4A, WMA, Ogg Vorbis, FLAC, AAC, and Bonk format support, integrates freedb/CDDB, CDText and ID3v2 tagging and is available in several languages.
[ Download fre:ac – free audio converter ]


Synology Open Source Project

Open source projects that are included with Synology DiskStation/RackStation series. The license used by these projects are different. Please refer to the LICENSE / COPYING / COPYRIGHT file inside each project or any announcement in source code.
[ Download Synology Open Source Project ]


Battle for Wesnoth Android Port

This is an unofficial android port of the PC game Battle for Wesnoth. Battle for Wesnoth is a turn-based fantasy strategy game, featuring many addictive campaigns, lots of units, different races, AI controlled players, multiplayer gaming and much more.
[ Download Battle for Wesnoth Android Port ]


Freeciv

Freeciv is a free turn-based multiplayer strategy game, in which each player becomes the leader of a civilization, fighting to obtain the ultimate goal: to become (or subvert) the greatest civilization.
[ Download Freeciv ]

Categories: Open Source

Apache OpenOffice Reaches Record Mark of 200 Million Downloads

SourceForge.net: Front page news - Sat, 12/03/2016 - 06:28

Apache OpenOffice reached a milestone this week, gaining a record mark of over 200 million downloads. It’s an achievement that, according to Apache OpenOffice Vice President Marcus Lange “is an acknowledgment of the previous work and a great motivation for the future.”

Recently the project experienced a high demand for their most current release 4.1.3, which was released just last October. Lange mentions in the official blog post that there are still many ways to distribute Apache OpenOffice, but that it is safe to say that Apache OpenOffice has one of the highest user bases in the world of free software projects. For that, he is immensely grateful to the users and everyone who has supported them thus far.

SourceForge has long been a proud partner of the Apache OpenOffice community, and in support of their great accomplishment, we’ve decided to add Apache OpenOffice to our list of featured projects. For the entire week next week, you’ll find Apache OpenOffice right on our homepage. It’s our way of saying congratulations, and here’s to more great milestones to come!

Download Apache OpenOffice, or visit their blog to know more.

Categories: Open Source

Announcing OSS-Fuzz: Continuous fuzzing for open source software

Google Open Source Blog - Thu, 12/01/2016 - 18:00
We are happy to announce OSS-Fuzz, a new Beta program developed over the past years with the Core Infrastructure Initiative community. This program will provide continuous fuzzing for select core open source software.

Open source software is the backbone of the many apps, sites, services, and networked things that make up “the internet.” It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it.

Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers. That's where fuzz testing comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly.

In recent years, several efficient general purpose fuzzing engines have been implemented (e.g. AFL and libFuzzer), and we use them to fuzz various components of the Chrome browser. These fuzzers, when combined with Sanitizers, can help find security vulnerabilities (e.g. buffer overflows, use-after-free, bad casts, integer overflows, etc), stability bugs (e.g. null dereferences, memory leaks, out-of-memory, assertion failures, etc) and sometimes even logical bugs.

OSS-Fuzz’s goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution. OSS-Fuzz combines various fuzzing engines (initially, libFuzzer) with Sanitizers (initially, AddressSanitizer) and provides a massive distributed execution environment powered by ClusterFuzz.
Early successesOur initial trials with OSS-Fuzz have had good results. An example is the FreeType library, which is used on over a billion devices to display text (and which might even be rendering the characters you are reading now). It is important for FreeType to be stable and secure in an age when fonts are loaded over the Internet. Werner Lemberg, one of the FreeType developers, was an early adopter of OSS-Fuzz. Recently the FreeType fuzzer found a new heap buffer overflow only a few hours after the source change:

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa
READ of size 2 at 0x615000000ffa thread T0
SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)
   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31

OSS-Fuzz automatically notified the maintainer, who fixed the bug; then OSS-Fuzz automatically confirmed the fix. All in one day! You can see the full list of fixed and disclosed bugs found by OSS-Fuzz so far.
Contributions and feedback are welcomeOSS-Fuzz has already found 150 bugs in several widely used open source projects (and churns ~4 trillion test cases a week). With your help, we can make fuzzing a standard part of open source development, and work with the broader community of developers and security testers to ensure that bugs in critical open source applications, libraries, and APIs are discovered and fixed. We believe that this approach to automated security testing will result in real improvements to the security and stability of open source software.

OSS-Fuzz is launching in Beta right now, and will be accepting suggestions for candidate open source projects. In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure, a general heuristic that we are intentionally leaving open to interpretation at this early stage. See more details and instructions on how to apply here.

Once a project is signed up for OSS-Fuzz, it is automatically subject to the 90-day disclosure deadline for newly reported bugs in our tracker (see details here). This matches industry’s best practices and improves end-user security and stability by getting patches to users faster.

Help us ensure this program is truly serving the open source community and the internet which relies on this critical software, contribute and leave your feedback on GitHub.

By Mike Aizatsky, Kostya Serebryany (Software Engineers, Dynamic Tools); Oliver Chang, Abhishek Arya (Security Engineers, Google Chrome); and Meredith Whittaker (Open Research Lead).
Categories: Open Source

December 2016, “Community Choice” Project of the Month – ReactOS

SourceForge.net: Front page news - Thu, 12/01/2016 - 06:15

For our December “Community Choice” Project of the Month, the community elected ReactOS, an operating system based on the best Windows NT design principles. The team behind the project shared their thoughts about the project’s history, purpose, and direction.

SourceForge (SF): What made you start this project?
ReactOS Team (RT): ReactOS was started by a group of developers who, while impressed by the NT architecture of Microsoft’s Windows family, desired a more open development environment. They felt that not only would such an environment be beneficial to all developers that target Windows, providing insight into just how the underlying system actually works, but also provide a means to improve on the security and stability of the system by letting more people participate in its development.

SF: Has the original vision been achieved?
RT: We are still some ways to achieving complete application and driver compatibility with the NT5 family.

SF: Who can benefit the most from your project?
RT: Software developers seeking to understand how Windows works under the hood, OS developers/hobbyists who want an example of a non-Unix style OS, and users who require an NT5 era Windows platform for application or hardware compatibility, and who want continued updates for security and the like.

SF: What core need does ReactOS fulfill?
RT: With Microsoft having end-of-lifed the NT5 family, including XP and 2003, users who would prefer that environment, or a much more lightweight Windows environment, would be better served by ReactOS when it is completed.

SF: What’s the best way to get the most out of using ReactOS?
RT: Try it out on VirtualBox or VMware, and don’t go in expecting everything to be perfect yet. If you’re ready for some tinkering, or excited about the ability to customize everything, you’ll quickly find things to try or areas to take a deeper look at.

SF: What has your project team done to help build and nurture your community?
RT: Probably the biggest effort to help build our Community started a couple [of] years ago when we defined a new Product-Community strategy. As our first milestone we launched a crowdfunding campaign letting the Community decide which apps they wanted to see start working in ReactOS. The main objective of this move was not just to raise some funding for such on-demand development but also striking news, reaching new users, showing them their opinion counts, and helping them to find that lot of software was already working. This, as planned, enlarged our Community but, even more important, it helped build a closer relationship with them. The new site released at the same time, and largely requested by the Community, has been proven to improve the first overall impression of the ReactOS product itself.
Since then, and as part of our second milestone, we created several scripts which share in our social channels a constant flow of information from all our services. The best fixes from our Jira bugtracker, the latest videos from our ReactOS Youtube channel, blog posts created by developers or official news from the ReactOS website are, among others, shared now automatically. Now the Community can track how ReactOS is evolving daily and interact in real time with it. These interactions help attract new members to the Community and enhance a closer relationship with the current ones. Nowadays we’re probably one of the most complete in sharing infra among the open source projects out there.

SF: Have you all found that more frequent releases helps build up your community of users?
RT: Very much so, we’ve increased our release frequency considerably this year and have seen a far greater influx of new users and testers on our forum. Word of mouth is our only advertising way and seems to combine perfectly with a faster release tempo.
At the same time, our current 3 months lapse helps to feel the difference in terms of stability and compatibility within releases.

SF: What was the first big thing that happened for your project?
RT: We’ve had quite a few milestones in ReactOS’ history. The first time the OS booted by itself instead of being bootstrapped by DOS, the first time we were able to switch between two windowed applications, the first time network communications worked- there were a lot of big things over the course of the project.

SF: What helped make those happen?
RT: All of the milestones we’ve achieved have been due to the hard work and very often tenacity of our developers and testers. Some of these guys had to get very creative in solving the problems they encountered in trying to develop an OS from scratch. Also we can’t thank enough the support of our donors. Since we don’t have any company supporting our development, they are the ones helping to hire new developers and paying our server bills.

SF: How has SourceForge and its tools helped your project reach that success?
RT: All of our releases are done through SourceForge, helping us to reduce the cost of distribution of our product. The metrics that we get from the downloads provide us with a good idea of where our prospective user base is from.
These metrics help us to understand our users’ behaviour: Are they willing to test bootcds or livecds? Are we attracting more users release after release? How does it affect the downloads, a faster release cycle as we’re doing now? How does a particular marketing action done affect the ReactOS downloads? How is the inertia (download of old releases) evolving?
But also it helps us to predict the expected visitors in our website for the next releases so we can ensure the needed resources for the peak days.
Handling and analyzing correctly this data proves to be an amazing way to discover the health of the project and summed to the rest of our analytics helps to draw a roadmap of our actions.

SF: What is the next big thing for ReactOS?
RT: There are several next big things coming pretty soon. The first one is the integration of the results from this year’s Google Summer of Code. Also we’re working hard on having Word 2010, Java RE and Google Chrome supported, since they are the apps selected by our Community in the IndieGoGo crowdfunding campaign.

SF: Do you have the resources you need to make those happen?
RT: So far we’ve been fairly fortunate this year, contributions and manpower have been quite steady and we expect to get the improvements in without too much fuss. With that said, ReactOS is on its way to reach Beta status. Beta supposes a jump in quality and for such we’ll need to place full time developers to reach it. Reaching Beta is not as costly as one may think but some extra resources will be needed.

SF: If you had to do it over again, what would you do differently for ReactOS?
RT: ReactOS took a few shortcuts in its early days to try to achieve as many user-visible improvements as possible. Those hacks have been the source of considerable headaches as the team implemented more functionality correctly.
Looking back, we probably should have fought that particular temptation and done the software engineering right the first time around.

SF: Is there anything else we should know?
RT: ReactOS is now hiring!. Thanks to the donations and contributions from our Community we’re planning to hire a new developer. So if you are skillful in Windows APIs or you are willing to help us in fixing bugs, you can just drop an email here.
Feel free to follow the progress of our current hired developer, Hermès, through his blog posts, discover what’s coming in the next release, or join the Community in Twitter, Facebook or Telegram.

[ Download ReactOS ]

Categories: Open Source

Docker + Dataflow = happier workflows

Google Open Source Blog - Wed, 11/30/2016 - 18:00
When I first saw the Google Cloud Dataflow monitoring UI -- with its visual flow execution graph that updates as your job runs, and convenient links to the log messages -- the idea came to me. What if I could take that UI, and use it for something it was never built for? Could it be connected with open source projects aimed at promoting reproducible scientific analysis, like Common Workflow Language (CWL) or Workflow Definition Language (WDL)?
Screenshot of a Dockerflow workflow for DNA sequence analysis.
In scientific computing, it’s really common to submit jobs to a local high-performance computing (HPC) cluster. There are tools to do that in the cloud, like Elasticluster and Starcluster. They replicate the local way of doing things, which means they require a bunch of infrastructure setup and management that the university IT department would otherwise do. Even after you’re set up, you still have to ssh into the cluster to do anything. And then there are a million different choices for workflow managers, each unsatisfactory in its own special way.

By day, I’m a product manager. I hadn’t done any serious coding in a few years. But I figured it shouldn’t be that hard to create a proof-of-concept, just to show that the Apache Beam API that Dataflow implements can be used for running scientific workflows. Now, Dataflow was created for a different purpose, namely, to support scalable data-parallel processing, like transforming giant data sets, or computing summary statistics, or indexing web pages. To use Dataflow for scientific workflows would require wrapping up shell steps that launch VMs, run some code, and shuttle data back and forth from an object store. It should be easy, right?

It wasn’t so bad. Over the weekend, I downloaded the Dataflow SDK, ran the wordcount examples, and started modifying. I had a “Hello, world” proof-of-concept in a day.

To really run scientific workflows would require more, of course. Varying VM shapes, a way to pass parameters from one step to the next, graph definition, scattering and gathering, retries. So I shifted into prototyping mode.

I created a new GitHub project called Dockerflow. With Dockerflow, workflows can be defined in YAML files. They can also be written in pretty compact Java code. You can run a batch of workflows at once by providing a CSV file with one row per workflow to define the parameters.

Dataflow and Docker complement each other nicely:

  • Dataflow provides a fully managed service with a nice monitoring interface, retries,  graph optimization and other niceties.
  • Docker provides portability of the tools themselves, and there's a large library of packaged tools already available as Docker images.

While Dockerflow supports a simple YAML workflow definition, a similar approach could be taken to implement a runner for one of the open standards like CWL or WDL.

To get a sense of working with Dockerflow, here’s “Hello, World” written in YAML:

defn:
  name: HelloWorkflow
steps:
- defn:
    name: Hello
    inputParameters:
      name: message
      defaultValue: Hello, World!
    docker:
      imageName: ubuntu
      cmd: echo $message

And here’s the same example written in Java:

public class HelloWorkflow implements WorkflowDefn {
  @Override
  public Workflow createWorkflow(String[] args) throws IOException {
    Task hello =
        TaskBuilder.named("Hello").input("message", “Hello, World!”).docker(“ubuntu”).script("echo $message").build();
    return TaskBuilder.named("HelloWorkflow").steps(hello).args(args).build();
  }
}

Dockerflow is just a prototype at this stage, though it can run real workflows and includes many nice features, like dry runs, resuming failed runs from mid-workflow, and, of course, the nice UI. It uses Cloud Dataflow in a way that was never intended -- to run scientific batch workflows rather than large-scale data-parallel workloads. I wish I’d written it in Python rather than Java. The Dataflow Python SDK wasn’t quite as mature when I started.

Which is all to say, it’s been a great 20% project, and the future really depends on whether it solves a problem people have, and if others are interested in improving on it. We welcome your contributions and comments! How do you run and monitor scientific workflows today?

By Jonathan Bingham, Google Genomics and Verily Life Sciences
Categories: Open Source

Introducing HTTPS for Project Websites

SourceForge.net: Front page news - Wed, 11/30/2016 - 17:54

We are very excited to offer HTTPS web hosting to all projects. With a single click, projects can opt-in to switch their web hosting from http://name.sourceforge.net to https://name.sourceforge.io Project admins can find this option in the Admin page, under “HTTPS”, naturally

Categories: Open Source

Google Summer of Code 2016 wrap-up: STE||AR

Google Open Source Blog - Tue, 11/29/2016 - 18:00
This is part of a series of guest posts from students, mentors and organization administrators who participated in Google Summer of Code (GSoC) 2016. GSoC is an annual program which pairs university students with mentors to work on open source software.


This summer the STE||AR Group was proud to mentor four students through Google Summer of Code. These students worked on a variety of projects which helped improve our software, HPX. This library is a distributed C++ runtime system which supports a standards compliant API and helps users scale their applications across thousands of machines.

The improvements to the code base will help our team and users of HPX around the world. A summary of our students’ projects:

Parsa Amini – HPX Debugger

Developing a better distributed debugging tool is essential to increase the programmability of HPX. Parsa’s project, Scimitar, aims to facilitate the debugging process for HPX programmers by extending the features of GDB, an existing debugger. The project then complements it with new commands for easier switching between localities across clusters, HPX thread debugging, awareness of internal HPX data structures, and semi-automated preparation for distributed debugging sessions. Additional functionality such as locating an object and viewing the queue information on each core is provided through using API provided by HPX itself. His work can be found on GitHub.

Aalekh Nigam – Implement a Map/Reduce Framework

This project aimed to expose a Map/Reduce programming model over HPX. During the summer, Aalekh was able to develop a single node implementation of HPXflow (map/reduce programming model) and laid the groundwork for the further multi-node version with database support. Although the initial task was limited to implementing the Map/Reduce model, he was also able to implement an improved dataflow model as well.

Minh-Khanh Do - Working on Parallel Algorithms for HPX::Vector

Minh-Khanh’s task was to take the parallel algorithms and add the functionality required to work on the segmented hpx::vector. Under his mentor John Biddscombe, he implemented the segmented_fill algorithm, which was successfully merged into the main codebase. Additionally, Minh-Khanh implemented the segmented_scan algorithm which includes inclusive and exclusive_scan. These changes are included in a pull request and have been merged. Using the segmented scan algorithm it is possible to perform tasks such as evaluating polynomials and to implement other algorithms such as quicksort.

Satyaki Upadhyay - Plugin Mechanism for thread schedulers in HPX

In HPX, schedulers are statically linked and must be built at compile-time. Satyaki’s project involved converting this statically linked scheme into a plugin system which would allow arbitrary schedulers to be dynamically loaded. These changes bring several benefits. They provide a layer of abstraction and follow the open/closed principle of software design as well as allowing developers to write their own custom schedulers while conforming to a uniform API. The project proceeded in two steps. The first involved the creation of plugin modules of the schedulers and registering them with HPX. The second step was to implement the loading and subsequent use of the chosen scheduler.

We would like to thank our students and mentors for the time that they have contributed to HPX this summer. In addition, we would like to thank Google for the opportunity that they provided the STE||AR Group to work with developers around the globe as well as the ability for students to interact with vibrant open source projects worldwide.

By Adrian Serio, Organization Administrator for The STE||AR Group
Categories: Open Source

It’s that time again: Google Code-in starts today!

Google Open Source Blog - Mon, 11/28/2016 - 21:18
Today marks the start of the 7th year of Google Code-in (GCI), our pre-university contest introducing students to open source development. GCI takes place entirely online and is open to students between the ages of 13 and 17 around the globe.
The concept is simple: complete bite-sized tasks (at your own pace) created by 17 participating open source organizations on topic areas you find interesting:

  • Coding
  • Documentation/Training
  • Outreach/Research
  • Quality Assurance
  • User Interface

Tasks take an average of 3-5 hours to complete and include the guidance of a mentor to help along the way. Complete one task? Get a digital certificate. Three tasks? Get a sweet Google t-shirt. Finalists get a hoodie. Grand Prize winners get a trip to Google headquarters in California.

Over the last 6 years, 3213 students from 99 countries have successfully completed tasks in GCI. Intrigued? Learn more about GCI by checking out our rules and FAQs. And please visit our contest site and read the Getting Started Guide.

Teachers, if you are interested in getting your students involved in Google Code-in you can find resources here to help you get started.

By Mary Radomile, Open Source Programs Office
Categories: Open Source

Projects of the Week, November 28, 2016

SourceForge.net: Front page news - Mon, 11/28/2016 - 06:35

Here are the featured projects for the week, which appear on the front page of SourceForge.net:

LibreCAD

LibreCAD is an open-source cross-platform 2D CAD program, which is translated in over 30 languages. http://translate.librecad.org LibreCAD is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. License: https://www.gnu.org/licenses/gpl-2.0.html Source: https://github.com/LibreCAD/LibreCAD/releases Track the latest development: https://github.com/LibreCAD/LibreCAD/milestones https://github.com/LibreCAD/LibreCAD/commits/master Create bug reports and feature requests: https://github.com/LibreCAD/LibreCAD/issues Get help: – http://forum.librecad.org – http://wiki.librecad.org
[ Download LibreCAD ]


slackel

Slackel is a Linux distribution based on Slackware and Salix. It is fully compatible with Slackware and Salix but the difference is that it includes the current version of Slackware. So Slackware users can benefit from Slackel repositories. It is available in three editions, KDE, Openbox and Fluxbox. Slackel disc images are offered in two different forms, Installation disc image and Live disc image. Slackel is developed in Greece by Dimitris Tzemos.
[ Download slackel ]


Wine

Wine is an Open Source implementation of the Windows API on top of X and Unix. Wine provides both a development toolkit for porting Windows sources to Unix and a program loader, allowing many unmodified Windows binaries to run on x86-based Unixes.
[ Download Wine ]


DisplayCAL

DisplayCAL (formerly known as dispcalGUI) is a graphical user interface for the display calibration and profiling tools of Argyll CMS, an open source color management system. Calibrate and characterize your display devices using one of the many supported measurement instruments, with support for multi-display setups and a variety of available settings like customizable whitepoint, luminance, tone response curve as well as the option to create accurate look-up-table ICC profiles as well as some proprietary 3D LUT formats. Check the accuracy of profiles and 3D LUTs via measurements.
[ Download DisplayCAL ]


SynWrite add-ons

Collection of add-ons for SynWrite editor.
[ Download SynWrite add-ons ]


plantuml

PlantUml allows to quickly create some UML diagram using a simple textual description language.
[ Download plantuml ]


TrouSerS

TrouSerS – An open-source TCG Software Stack implementation.
[ Download TrouSerS ]


Berryboot Updated OS Images

This is the biggest list of OS images for Berryboot. All these images have been tested before public availability. The original OS images were downloaded from their respective official websites, I did not modified any data of the original nor converted OS images. I convert most of the OS images by using this method: http://www.berryterminal.com/doku.php/berryboot/adding_custom_distributions. RetroPie and others already have OS images for Berryboot, I just compress it and serve as a mirror. Fedora OS images were converted by using this tool: https://github.com/sjenning/rpi2-fedora-image-builder. I’m not part of the Berryboot project, I’m just a web developer with a Raspberry Pi 2. I cannot offer full support on these images. However, if you find any issue, please contact me and I’ll look into it. You can request OS images at: http://berryboot.alexgoldcheidt.com/request-os-images/
[ Download Berryboot Updated OS Images ]


VeraCrypt

VeraCrypt is a free disk encryption software brought to you by IDRIX (https://www.idrix.fr) and based on TrueCrypt 7.1a. It adds enhanced security to the algorithms used for system and partitions encryption making it immune to new developments in brute-force attacks. It also solves many vulnerabilities and security issues found in TrueCrypt. This enhanced security adds some delay ONLY to the opening of encrypted partitions without any performance impact to the application use phase. This is acceptable to the legitimate owner but it makes it much harder for an attacker to gain access to the encrypted data. All released files are PGP signed with key ID=0x54DDD393, available on key servers and downloadable at https://www.idrix.fr/VeraCrypt/VeraCrypt_PGP_public_key.asc VeraCrypt can mount TrueCrypt volumes. It also can convert them to VeraCrypt format. Documentation: https://veracrypt.codeplex.com/documentation FAQ : https://veracrypt.codeplex.com/wikipage?title=FAQ
[ Download VeraCrypt ]

Categories: Open Source

JHipster IDE

Date Created: Sat, 2016-11-26 14:43Date Updated: Mon, 2016-11-28 13:28Submitted by: Serano Colameo

JHipster-IDE is an Xtext DSL which provides texual editing support of JHipster Domain Language files for pupular IDEs and editors such as Eclipse, IDEA and Visual Studio Code. Thanx to Xtext, JHipster-IDE provides the features we have come to expect from modern IDE editors.

For more information please refer to our web page at JHipster IDE.

Categories: Open Source

LangLocker

Date Created: Sat, 2016-11-26 10:11Date Updated: Mon, 2016-11-28 09:13Submitted by: Andrey Mogilev

An Eclipse plugin which blocks unwanted input language switches (like English to Russian) while working in IDE.

In order to lock the current input language, run the 'Lock/unlock input language' command, or click on the corresponding button on the top panel, which looks like a lock with 'EN' characters on it. In order to unlock, do the same thing again.

Categories: Open Source

Registration Open | Eclipse Converge

Eclipse News - Fri, 11/25/2016 - 20:30
Discover the newest Eclipse Community event. Join us in San Jose alongside Devoxx US.
Categories: Open Source

FOSDEM PGDay 2017 & FOSDEM Devroom Call for Papers

PostgreSQL News - Fri, 11/25/2016 - 01:00

FOSDEM PGDay 2017 is a one day conference held this year on February 3rd followed immediately by the two day FOSDEM conference in Brussels at which we'll be hosting additional sessions on February 5th.

For more information on FOSDEM PGDay, please see:

https://2017.fosdempgday.org/

For more information on FOSDEM, please see:

https://fosdem.org/2017/

The call for papers for both the PGDay and FOSDEM Devroom has now been extended until midnight CET, 4th December. If you would like to submit a talk, please see the website:

https://2017.fosdempgday.org/callforpapers/

Talks may be on any topic related to PostgreSQL in some way. Suggested topic areas include:

  • Developing applications with or for PostgreSQL
  • Administering large scale PostgreSQL installations
  • Case studies and/or success stories of PostgreSQL deployments (or interesting failures)
  • PostgreSQL tools and utilities
  • PostgreSQL internals hacking
  • Community and local user groups
  • Tuning and performance improvements
  • Migration from other database systems
  • Scaling, Clustering and High Availability
  • Replication
  • Recovery and Backup strategies
  • Benchmarking and hardware
  • PostgreSQL related products
  • DevOps and continuous deployment/configuration/integration around PostgreSQL

Of course, we're happy to receive proposals for talks on other PostgreSQL related topics as well.

NOTE: Please submit talks through our website for both the PGDay and FOSDEM Devroom. DO NOT use the FOSDEM Pentabarf system as we are notmanaging the PostgreSQL talks though it.

The PostgreSQL Europe team will also have a table throughout the FOSDEM event; please feel free to stop by to chat with us about PostgreSQL and pickup some swag!

Categories: Database, Open Source

Open IoT Challenge 3.0 | Extended Deadline

Eclipse News - Thu, 11/24/2016 - 19:12
The submission deadline for the Open IoT Challenge 3.0 has been extended to November 30 @ 11:59 PM PT.
Categories: Open Source

Source Code Translation in Eclipse

Date Created: Thu, 2016-11-24 07:59Date Updated: Mon, 2016-12-05 09:43Submitted by: dtestehan

Every now and then a developer has to work on a code base that is in a language that is unknown to him. In this case, he would need to copy variable, class, method names to Google Translate to see what they actually mean.

The Source Code Translation in Eclipse tries to help such developers, by providing a popup with the translated words, when hovering the mouse over them. For the plugin to work, a properties file containing the word to be translated, and the translation needs to be created and added from the Preferences section of the plugin.

For help/suggestions contact me at tdan89@yahoo.com

Dan Testehan

Categories: Open Source