Skip to content

Software Development News: .NET, Java, PHP, Ruby, Agile, Databases, SOA, JavaScript, Open Source

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Open Source

Performance Comparison: Apache vs. IIS

DevX: Open Source Articles - Tue, 02/21/2017 - 17:09
There has been a lot of discussion regarding which of the two most popular Web servers is better. Take a look at this comparison of their characteristics and performance.
Categories: Open Source

Projects of the Week, February 20, 2017

SourceForge.net: Front page news - Mon, 02/20/2017 - 06:21

Here are the featured projects for the week, which appear on the front page of SourceForge.net:

Manjaro Community Torrents

This project is for download the Manjaro Officials and Community releases using a bittorrent client (console or graphical)
[ Download Manjaro Community Torrents ]


iDempiere

iDempiere = OSGi + ADempiere iDempiere Business Suite ERP/CRM/SCM done the community way. Focus is on the Community that includes Subject Matter Specialists, Implementors and End-Users. iDempiere is based on original Compiere/Adempiere plus a new architecture to use state-of-the-art technologies like OSGi, Buckminster, zk.
[ Download iDempiere ]


BibDesk

BibDesk is a graphical bibTeX bibliography manager for Mac OS X.
[ Download BibDesk ]


wl500g

wl500g is ASUS Oleg’s based custom firmware. It is free with custom firmware features, and supports the following branches and models: 1.9.2.7-rtn (mainstream, 2.6.22 kernel based, supports: > ASUS RT-N16, RT-N15U, RT-N12, RT-N12B1, RT-N10, RT-N10U, RT-N10UB1, WL-500gP, WL-500gPv2, WL-500W); 1.9.2.7-d (stable/suspended, 2.4.37 kernel based, supports: > ASUS WL-500gP, WL-500gPv2, WL-500W, WL-500gX, WL-520gU, WL-550gE, WL-330gE, WL-320gE, WL-320gP).
[ Download wl500g ]


simutrans

Simutrans is a cross-platform simulation game where players try to successfully manage transportation systems between places by land, air, and water for passengers, mail, and goods. Planes, ships, trains, trams, trucks, buses, or monorails are at your disposal, but factories operate based on contracts and passengers can only travel to their set destinations.
[ Download simutrans ]


strace

strace is a diagnostic, debugging and instructional userspace tracer for Linux. It is used to monitor and tamper with interactions between userspace processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. The operation of strace is made possible by the kernel feature known as ptrace.
[ Download strace ]


gretl

gretl is a cross-platform software package for econometric analysis, written in the C programming language.
[ Download gretl ]


berryboot

Berryboot is a simple operating system installer and boot selection screen for ARM devices such as the Raspberry Pi and Cubieboard. It allows you to put multiple Linux distribution on a single SD card.
[ Download berryboot ]


GeoServer

GeoServer is an open source software server written in Java that allows users to share and edit geospatial data. Designed for interoperability, it publishes data from any major spatial data source using open standards: WMS, WFS, WCS, WPS and REST
[ Download GeoServer ]

Categories: Open Source

JDBC 42.0.0 Released

PostgreSQL News - Mon, 02/20/2017 - 01:00

The JDBC group is proud to release a new version, and in keeping with the version renumbering meme we have released version 42.0.0.

Notable changes include:

  • Support for PostgreSQL versions below 8.2 was dropped
  • java.util.logging is now used for logging: logging documentation
  • Ensure executeBatch() can be used with pgbouncer. Previously pgjdbc could use server-prepared statements for batch execution even with prepareThreshold=0 (see issue 742)
  • Replication protocol API was added: replication API documentation, PR#550
  • Version bumped to 42.0.0 to avoid version clash with PostgreSQL version
  • Error position is displayed when SQL has unterminated literals, comments, etc (see issue 688)
Categories: Database, Open Source

Eclipse Newsletter - Top Eclipse Marketplace Plugins

Eclipse News - Fri, 02/17/2017 - 17:52
What are the top Eclipse Marketplace plugins? How can you benefit from them? Find out!
Categories: Open Source

Black Duck Hub

Date Created: Fri, 2017-02-17 11:04Date Updated: Wed, 2017-03-29 16:32Black Duck SoftwareSubmitted by: Black Duck Software

Version: 1.0.0

The Black Duck Hub plugin for Eclipse uses your Black Duck Hub instance to provide an overview of all dependencies in your Maven and Gradle projects. In addition, the plugin enables you to drill down by opening the informational page for any component/version it finds in the Hub.

Required: Black Duck Hub

Categories: Open Source

Find Which Port Is Being Used by a Process

DevX: Open Source Articles - Fri, 02/17/2017 - 15:33
Language: Enterprise||Expertise: Advanced
Categories: Open Source

Small but Significant Changes to Project Pages

SourceForge.net: Front page news - Fri, 02/17/2017 - 06:03

Continuous improvement is something that drives us here at SourceForge. We’re always looking for ways to improve our site and service in ways both big and small. In the past couple of months, we’ve made a few small but important changes that we think projects can really benefit from. These changes include:

  • HTTPS for git and svn code checkouts – Now all project developers can make use of the more secure https protocol.
  • HTTPS on all our download mirrors
  • High resolution project icons – Large, high resolution icons can now be uploaded to project pages. While these icons may look the same on your desktop, they will appear a lot clearer and better on phone screens and Retina/hi-dpi laptops, and when zooming in on a page.
  • Support for newer, more secure SSH keys – SourceForge now supports the new ed25519 and ecdsa formats.

We hope everyone will find these new enhancements useful and beneficial. Should you have any inquiries or issues regarding these changes, or would like to request other enhancements, please refer to our support page. We’ll do our best to make sure you continue to have a positive experience with SourceForge.

Categories: Open Source

Announcing TensorFlow 1.0

Google Open Source Blog - Thu, 02/16/2017 - 18:34
Originally posted on the Google Developer Blog

In just its first year, TensorFlow has helped researchers, engineers, artists, students, and many others make progress with everything from language translation to early detection of skin cancer and preventing blindness in diabetics. We're excited to see people using TensorFlow in over 6000 open source repositories online.

Today, as part of the first annual TensorFlow Developer Summit, hosted in Mountain View and livestreamed around the world, we're announcing TensorFlow 1.0:

It's faster: TensorFlow 1.0 is incredibly fast! XLA lays the groundwork for even more performance improvements in the future, and tensorflow.org now includes tips & tricksfor tuning your models to achieve maximum speed. We'll soon publish updated implementations of several popular models to show how to take full advantage of TensorFlow 1.0 - including a 7.3x speedup on 8 GPUs for Inception v3 and 58x speedup for distributed Inception v3 training on 64 GPUs!

It's more flexible: TensorFlow 1.0 introduces a high-level API for TensorFlow, with tf.layers, tf.metrics, and tf.losses modules. We've also announced the inclusion of a new tf.keras module that provides full compatibility with Keras, another popular high-level neural networks library.

It's more production-ready than ever: TensorFlow 1.0 promises Python API stability (details here), making it easier to pick up new features without worrying about breaking your existing code.

Other highlights from TensorFlow 1.0:
  • Python APIs have been changed to resemble NumPy more closely. For this and other backwards-incompatible changes made to support API stability going forward, please use our handy migration guide and conversion script.
  • Experimental APIs for Javaand Go
  • Higher-level API modules tf.layers, tf.metrics, and tf.losses - brought over from tf.contrib.learnafter incorporating skflowand TF Slim
  • Experimental release of XLA, a domain-specific compiler for TensorFlow graphs, that targets CPUs and GPUs. XLA is rapidly evolving - expect to see more progress in upcoming releases.
  • Introduction of the TensorFlow Debugger (tfdbg), a command-line interface and API for debugging live TensorFlow programs.
  • New Android demos for object detection and localization, and camera-based image stylization.
  • Installation improvements: Python 3 docker images have been added, and TensorFlow's pip packages are now PyPI compliant. This means TensorFlow can now be installed with a simple invocation of pip install tensorflow.
We're thrilled to see the pace of development in the TensorFlow community around the world. To hear more about TensorFlow 1.0 and how it's being used, you can watch the TensorFlow Developer Summit talks on YouTube, covering recent updates from higher-level APIs to TensorFlow on mobile to our new XLA compiler, as well as the exciting ways that TensorFlow is being used:

Click here for a link to the livestream and video playlist
(individual talks will be posted online later in the day).The TensorFlow ecosystem continues to grow with new techniques like Foldfor dynamic batching and tools like the Embedding Projector along with updatesto our existing tools like TensorFlow Serving. We're incredibly grateful to the community of contributors, educators, and researchers who have made advances in deep learning available to everyone. We look forward to working with you on forums like GitHub issues, Stack Overflow, @TensorFlow, the discuss@tensorflow.orggroup, and at future events.
By Amy McDonald Sandjideh, Technical Program Manager, TensorFlow

Categories: Open Source

Eclipse IoT Announces Support for OMA LightweightM2M 1.0 Device Management Standard

Eclipse News - Wed, 02/15/2017 - 15:11
We are pleased to announce the Eclipse Leshan project will support OMA's newly ratified LwM2M 1.0 standard.
Categories: Open Source

JDT Extension

Date Created: Wed, 2017-02-15 02:41Date Updated: Sat, 2017-02-18 03:22Yongqiang LiSubmitted by: Yongqiang Li

The motivation is to extend JDT and make JDT more powerful and easy-to-use.

Here is the function list.
* Guava-based hashCode(), equals() and toString()
* Search with Google, StackOverflow

See more details here (http://bloodlee.github.io/jdt_extension/)

Categories: Open Source

RTC Eclipse Client Add-Ons

Date Created: Tue, 2017-02-14 04:44Date Updated: Tue, 2017-02-14 09:16N/ASubmitted by: Fredrik Attebrant

Add-on to the IBM Rational Team Concert Eclipse client which adds:

- Copy workitem "Summary : ID", e.g. "Adds copy workitem summary followed by id : 12345"
The copied string is truncated to 72 characters making it fit nicely within e.g. a git commit comment.
Really useful when your organization requires the format "Summary : ID" instead of "ID : Summary" which
the RTC client has support for.

Install using update site: https://github.com/fredrikattebrant/se.attebrant.rtc.addons/raw/master/s...

Categories: Open Source

Registrations now open for pgDay Asia 2017

PostgreSQL News - Tue, 02/14/2017 - 01:00

Team pgDay Asia is happy to announce that registrations for the conference are now open. For a limited time, we are offering group discounts for 3+ people. Register yourself by clicking: http://2017.pgday.asia/#registration.

Schedule for the 2-day single-track conference has also been published: http://2017.pgday.asia/#schedule

Please note that the 2nd day of the conference is part of FOSSASIA's dedicated PostgreSQL track. Tickets for FOSSASIA can be purchased separately here: http://2017.fossasia.org/tickets/

A special shout out to all our speakers and sponsors. This event is not possible without your support. For anyone wishing to sponsor, please send an email to pgday-asia@googlegroups.com.

pgDay Asia 2017 is scheduled to be held in Singapore on 17th & 18th of March. More details here: http://2017.pgday.asia/. The conference will be held along with FOSSASIA - one of the largest FOSS conference on the planet. pgDay Asia will be PostgreSQL conference series for all PostgreSQL enthusiasts in the Asia Pacific region.

Categories: Database, Open Source

Projects of the Week, February 13, 2017

SourceForge.net: Front page news - Mon, 02/13/2017 - 06:38

Here are the featured projects for the week, which appear on the front page of SourceForge.net:

FileBot

FileBot is the ultimate tool for renaming your movies, tv shows or anime and downloading subtitles. It’s smart, streamlined for simplicity and just works. FileBot supports Windows, Linux and Mac, plus there’s a full-featured command-line interface for all sorts of automation.
[ Download FileBot ]


LibreCAD

LibreCAD is an open-source cross-platform 2D CAD program, which is translated in over 30 languages. http://translate.librecad.org LibreCAD is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. License: https://www.gnu.org/licenses/gpl-2.0.html Source: https://github.com/LibreCAD/LibreCAD/releases Track the latest development: https://github.com/LibreCAD/LibreCAD/milestones https://github.com/LibreCAD/LibreCAD/commits/master Create bug reports and feature requests: https://github.com/LibreCAD/LibreCAD/issues Get help: – http://forum.librecad.org – http://wiki.librecad.org
[ Download LibreCAD ]


npackd-logoNpackd

Npackd is a free Windows package manager that also serves as an installer and app store for Windows apps. With Npackd you can find and install software in minutes with absolutely no adware or other distractions. Apart from finding and installing software, it can keep your system updated and uninstall apps when no longer necessary. Installing and uninstalling apps is completely automated (silent or unattended installation and un-installation), making app management a breeze.
[ Download Npackd ]


antergos

The purpose of Antergos is to provide a modern, elegant and powerful operating system based on one of the best Linux distributions out there, Arch Linux. Antergos is easy to use and very customizable It is open source, free and based on the fast and lightweight Arch Linux. Antergos uses the official Arch Linux package repositories and the AUR (user-submitted packages) along with its own software repositories. As with other GNU/Linux systems, Antergos is virtually free of viruses and spyware
[ Download antergos ]


gnuplot

A famous scientific plotting package, features include 2D and 3D plotting, a huge number of output formats, interactive input or script-driven options, and a large set of scripted examples.
[ Download gnuplot ]


ApexDC++

ApexDC™ is an innovative Direct Connect client based on the DC++ and StrongDC++ projects. Direct Connect provides a rare blend of seamless file sharing and community based chat and messaging between users. ApexDC features plugins and scripting through Lua and much more. Both operators and users alike should find it a pleasant experience. Note: as of May 28th 2016 ApexDC is now primarily distributed as two separate installers one targetting 32bit and the other 64bit operating systems. Additionally XP compatible binaries are now only offered as a separate download from the files section.
[ Download ApexDC++ ]


Tcl

Tool Command Language (Tcl) is an interpreted language and very portable interpreter for that language. Tcl is embeddable and extensible, and has been widely used since its creation in 1988 by John Ousterhout. Bug reports to http://core.tcl.tk/tcl/ Follow code development at http://core.tcl.tk/tcl/
[ Download Tcl ]


Super Grub2 Disk

Super GRUB2 Disk helps you to boot into most any Operating System (OS) even if you cannot boot into it by normal means.
[ Download Super Grub2 Disk ]

Categories: Open Source

PostgreSQL@SCaLE15x - March 2-3, 2017, Pasadena Convention Center

PostgreSQL News - Mon, 02/13/2017 - 01:00
Please join us at PostgreSQL@SCaLE15x!

PostgreSQL@SCaLE is an embedded PostgreSQL conference held as part of SCaLE15x on Thursday and Friday, March 2-3, 2017. We will have 24 one-hour PostgreSQL talks -- two rooms for two days. This is by far the best value for your money of any North American PostgreSQL event!

For the talk schedule, please see: We will also have a PostgreSQL community booth in the exhibition hall all weekend.

SCaLE15x (the 15th annual Southern California Linux Expo), taking place on March 2-5, 2017 at the Pasadena Convention Center, is the largest community-run open-source and free software conference in North America. It is held annually in the Los Angeles area. PostgreSQL@SCaLE is always very popular and well attended. Don't miss out!

There is a nominal registration cost, but we have a 50% discount code: PSQL

Please go register here.

PostgreSQL@SCaLE15x is made possible by the United States PostgreSQL Association, a nonprofit 501(c)(3) created to support PostgreSQL in the United States through user group development, conferences, educational initiatives, and fun. There is no way we would be able to produce PostgreSQL@SCaLE15x without the generous support of our sponsors:

2ndQuadrant
Amazon Web Services
Command Prompt
OpenSCG
PostgreSQL Consulting

We hope to see everyone there!

The PostgreSQL@SCaLE15x Committee Joe, Jim, and Gabrielle

Categories: Database, Open Source

Open-sourcing Google Earth Enterprise

Google Open Source Blog - Fri, 02/10/2017 - 21:09
(originally posted on the Geo Developers blog)

We are excited to announce that we are open-sourcing Google Earth Enterprise (GEE), the enterprise product that allows developers to build and host their own private maps and 3D globes. With this release, GEE Fusion, GEE Server, and GEE Portable Server source code (all 470,000+ lines!) will be published on GitHub under the Apache2 license in March.
Screen Shot 2017-01-26 at 2.51.24 PM.pngOriginally launched in 2006, Google Earth Enterprise provides customers the ability to build and host private, on-premise versions of Google Earth and Google Maps. In March 2015, we announced the deprecation of the product and the end of all sales. To provide ample time for customers to transition, we have provided a two year maintenance period ending on March 22, 2017. During this maintenance period, product updates have been regularly shipped and technical support has been available to licensed customers.

Feedback is important to us and we’ve heard from our customers that GEE remains in-use in mission-critical applications. Many customers have not transitioned to other technologies. Open-sourcing GEE allows our customer community to continue to improve and evolve the project in perpetuity. Note that the implementations for Google Earth Enterprise Client, Google Maps JavaScript® API V3 and Google Earth API will not be open sourced. The Enterprise Client will continue to be made available and updated. However, since GEE Fusion and GEE Server are being open-sourced, the imagery and terrain quadtree implementations used in these products will allow third-party developers to build viewers that can consume GEE Server Databases.

We’re thankful for the help of our GEE partners in preparing the codebase to be migrated to GitHub. It’s a lot of work and we cannot do it without them. It is our hope that their passion for GEE and GEE customers will serve to lead the project into its next chapter.

Looking forward, GEE customers can use Google Cloud Platform (GCP) instead of legacy on-premises enterprise servers to run their GEE instances. For many customers, GCP provides a scalable and affordable infrastructure as a service where they can securely run GEE. Other GEE customers will be able to continue to operate the software in disconnected environments. However, we believe that the advantages of incorporating even some of the workloads on GCP will become apparent (such as processing large imagery or terrain assets on GCP that can be downloaded and brought to internal networks, or standing up user-facing Portable Globe Factories).

Moreover, GCP is increasingly used as a source for geospatial data. Google’s Earth Engine has made available over a petabyte of raster datasets which are readily accessible and available to the public on Google Cloud Storage. Additionally, Google uses Cloud Storage to provide data to customers who purchase Google Imagery today. Having access to massive amounts of geospatial data, on the same platform as your flexible compute and storage, makes generating high quality Google Earth Enterprise Databases and Portables easier and faster than ever.

We will be sharing a series of white papers and other technical resources to make it as frictionless as possible to get open source GEE up and running on Google Cloud Platform. We are excited about the possibilities that open-sourcing enables, and we trust this is good news for our community. We will be sharing more information when we launch the code in March on GitHub. For general product information, visit the Google Earth Enterprise Help Center. Review the essential and advanced training for how to use Google Earth Enterprise, or learn more about the benefits of Google Cloud Platform.

Posted by Avnish Bhatnagar, Senior Technical Solutions Engineer, Google Cloud
Categories: Open Source

ArchiMate for Papyrus

Date Created: Fri, 2017-02-10 07:15Date Updated: Fri, 2017-02-10 09:53Adocus ABSubmitted by: Thomas Wiman

The ArchiMate add-in is an open-source initiative founded by Adocus. The add-in enables you to model enterprise architecture models based upon the well recognized and popular ArchiMate® 2.1 notation by The Open Group.
The ArchiMate notation offers strategic modeling with motivation for the planned architecture initiative, identification of affected architectures such as business architecture, application architecture and technical architecture. ArchiMate does also support modeling of the implementation and migration between the current architecture state (plateau) to the next.
Key Features

  • Support for ArchiMate 2.1 in Papyrus 2.0.0
  • Presentation of both rectangular and symbol presentation of ArchiMate elements
  • Palettes (toolboxes) for accelerated modeling
  • Possibility for mixing ArchiMate and UML models to establish traceability between strategic and realization models
  • Model templates, one for a clean/empty model offering structure and one model with the ArchiSurance example from The Open Group
  • Graphic support for differentiation of the different extensions/layers in ArchiMate
  • Help pages
Categories: Open Source

Common Causes of Open Source Project Failure and How to Avoid Them

SourceForge.net: Front page news - Fri, 02/10/2017 - 06:20

One thing about open source is that even the failures contribute to the next thing that comes up. Unlike a company that could spend a million dollars in two years and fail and there’s nothing really to show for it, if you spend a million dollars on open source, you probably have something amazing that other people can build on. –Matt Mullenweg

In open source software development, one man’s failed project can be another man’s next successful project. As Matt Mullenweg has pointed out, failed projects can be the basis upon which other, more successful projects can be founded and developed.

But no matter how much of a boon one failed project may provide another, it’s still a loss for at least one person- the founder of the failed project. Between the person who spends a million dollars and fails and the person who spends the same and succeeds, you definitely don’t want to be the former! And the best way to avoid such failure is to see it coming.

Identifying the Common Causes of Failure

To see failure coming is to spot the causes. There are various causes of project failure, the most common being:

  • The developer loses interest. This loss of interest can stem from a number of things- the developer may have already stagnated with the project, is tired of the users’ demands, or is looking to focus on other opportunities. And without a maintainer to follow him, the project will inevitably be abandoned.
  • Lack of resources. Despite the “free” label, open source software development can take up a lot of resources: manpower, time and money. If there aren’t enough of these resources to support the project, abandonment will be likely.
  • Misunderstandings within the project team. If the project is worked on by people who disagree on a number of things, there is a high possibility of division among team members and the project being forked. If however the team is too fragmented and the misunderstanding too great, the project may not continue at all.
  • The project lagging behind. This could be in terms of technology, competition, or both. Technology is constantly evolving and with it, users’ expectations and demands. If a project doesn’t keep up or falls behind its competitors, it will eventually become obsolete.
  • The project becoming too big. Another possible cause of project failure is when it gets too big for the current developers and no one else steps in to help.
  • No standards or quality control. Be it in documentation, execution or support.
  • Legal problems. Very few open source projects can survive legal challenges, especially against large companies.

How to Avoid Them?

Sometimes, there’s just no avoiding the disintegration of a good project. When you see the causes coming however, there are still some things you can do to save the project at the most, and soften the blow of the loss at the very least.

  1. Ask for help. Asking for help from your community can solve a lot of project problems. When you start having difficulty managing the project on your own, you can ask the community for help maintaining it or ask them to fork it. If there’s a roadblock you can’t solve on your own, present it to the community and see if they can come up with a solution. If you need resources, the community can help you there too. Of course, this will require a great deal of humility and willingness to ask for assistance on your part.
  2. Innovate. If your project is stagnating, do what you need to do in order to keep it moving and keep the software improving. This may require additional work, research, learning new technologies and asking for help or input from other developers.
  3. Communicate. Miscommunication is a leading cause of misunderstanding, which in turn causes a host of problems for projects. Make sure you are able to openly exchange ideas with your team and listen to and share with your community as well.
  4. Always be aware. There may be more than one reason why your project is failing. Always be aware of the major factors affecting your project, and learn to accept when things are no longer within your control. Accepting that you may no longer be interested in a project or that it is becoming too big for you for instance, will allow you to seek help sooner and increase the chances of saving the project.
Categories: Open Source

Engage BlackVault Code Signing and Certification Authority Appliance

Date Created: Wed, 2017-02-08 19:11Date Updated: Thu, 2017-02-09 09:16EngageSubmitted by: Tim Dyck

Engage BlackVault cryptographic appliances make code signing and certification authority
operations secure.
Protect Cryptographic Keys and Operations
Perform Crypto Operations in Tamper Reactive Hardware

• Private / Public key pair generation, storage and backup
• Code Signing
• Certification Authority X.509 certificate generation, distribution and management
• FIPS 140 Level 3 Hardware Security Module ( HSM )
Due to continuously evolving threats (viruses, ransomware, Trojans, etc.), data security best practices dictate private keys used for code signing and certificates should never be “in the clear”. Unfortunately, too often these keys are stored on servers with weak password protection, and used in the clear by a server’s OS. This creates high risk for key theft or misuse, often resulting in disruption and financial harm to an organization.
Signing your Eclipse project code prior to publishing is a critical step in the DevOps process. With signed code users, can authenticate the author and verify the code wasn’t tampered with. Just as critical is keeping the private crypto keys and the operations they perform secure. The BlackVault makes this possible.
BlackVault

• Easily integrates with the Eclipse IDE
• Touch screen display for simple setup and use
• Zeroize (erases) keys when tampered with
• Online and Offline operation
• Code signing and certification authority applications
• Smart card Quorum for multi-department approvals

Categories: Open Source

PostgreSQL 9.6.2, 9.5.6, 9.4.11, 9.3.16 and 9.2.20 released!

PostgreSQL News - Thu, 02/09/2017 - 01:00

The PostgreSQL Global Development Group has released an update to all supported versions of our database system, including 9.6.2, 9.5.6, 9.4.11, 9.3.16, and 9.2.20. This release includes fixes that prevent data corruption issues in index builds and in certain write-ahead-log replay situations, which are detailed below. It also patches over 75 other bugs reported over the last three months.

Users should plan to apply this update at the next scheduled downtime.

Build corruption with CREATE INDEX CONCURRENTLY

There existed a race condition if CREATE INDEX CONCURRENTLY was called on a column that had not been indexed before, then rows that were updated by transactions running at the same time as the CREATE INDEX CONCURRENTLY command could have been indexed incorrectly.

If you suspect this may have happened, the most reliable solution is to rebuild affected indexes after installing this update.

This issue is present in the 9.2, 9.3, 9.4, 9.5, and 9.6 series of PostgreSQL.

Fixes for visibility and write-ahead-log stability

This release contains several fixes to improve the stability of visible data and WAL logging that we wish to highlight here.

Prior to this release, data could be prematurely pruned by a vacuum operation when a special snapshot used for catalog scans was presently available. Specifically, the vacuum operation would not be aware of the oldest xmin for this special snapshot. The error would surface with a message such as:

"cache lookup failed for relation 1255"

This release ensures that vacuum operations will account for the catalog scan snapshots.

Additionally, there are several fixes to improved the stability of write-ahead-logging, including:

  • A fix for BRIN index WAL logging where a replay could make a portion of the BRIN index useless and require recomputing
  • A fix for an unlogged table where a WAL-log would be created under the "wal_level = minimal" setting, and upon replay after crash, the table would not appear to be properly reset
  • A fix in the WAL page header validation when re-reading segments that fixes the error of "out-of-sequence TLI" that could be reported during recovery

These issues are present in the 9.6 series of PostgreSQL releases and may also be present in the 9.2, 9.3, 9.4, and 9.5 series.

Bug Fixes and Improvements

This update also fixes a number of bugs reported in the last few months. Some of these issues affect only the 9.6 series, but many affect all supported versions. There are more than 75 fixes provided in this release, including:

  • Several fixes for operating in hot standby mode
  • Disallow setting the num_sync field to zero in synchronous_standby_names
  • Don't count background worker processes against a user's connection limit
  • Fix check for when an extension member object can be dropped
  • Fix tracking of initial privileges for extension member objects so that it works correctly with ALTER EXTENSION ... ADD/DROP
  • Several vacuum and autovacuum fixes
  • Fix CREATE OR REPLACE VIEW to update the view query before attempting to apply the new view options
  • Make sure ALTER TABLE preserves index tablespace assignments when rebuilding indexes
  • Several fixes to the query planner, including fixes for foreign tables and CTEs
  • Several fixes around the full-text search feature to improve accuracy and performance of the search
  • Several fixes and performance improvements in several array functions
  • Several fixes around the interaction of foreign-key constraints with trigger functions around specific ALTER TABLE operations
  • Remove optimizations for date/time data types that returned incorrect data
  • Fix incorrect use of view reloptions as regular table reloptions
  • Fix incorrect "target lists can have at most N entries" complaint when using ON CONFLICT with wide tables
  • Fix spurious "query provides a value for a dropped column" errors during INSERT or UPDATE on a table with a dropped column
  • Prevent multicolumn expansion of foo.* in an UPDATE source expression
  • Ensure that column typmods are determined accurately for multi-row VALUES constructs
  • Several fixes for the psql command-line tool
  • Prevent multiple calls of pg_start_backup() and pg_stop_backup() from running concurrently
  • Several fixes for pg_dump, pg_restore, and pg_basebackup, including a possible pg_basebackup failure on standby server when including WAL files
  • Several fixes for parallel workers and plans for parallel queries including fixing a crash if the number of workers available to a parallel query decreases during a rescan
  • Several fixes to PL/pgSQL, PL/Python, and PL/Tcl
  • Several fixes for contrib modules
  • Allow DOS-style line endings in ~/.pgpass files, even on Unix

Update time zone data files to tzdata release 2016j for DST law changes in northern Cyprus (adding a new zone Asia/Famagusta), Russia (adding a new zone Europe/Saratov), Tonga, and Antarctica/Casey. Historical corrections for Italy, Kazakhstan, Malta, and Palestine. Switch to preferring numeric zone abbreviations for Tonga.

Updating

All PostgreSQL update releases are cumulative. As with other minor releases, users are not required to dump and reload their database or use pg_upgrade in order to apply this update release; you may simply shut down PostgreSQL and update its binaries.

If you believe you have been affected by the aforementioned CREATE INDEX CONCURRENTLY bug, you will have to rebuild the index. Here is an example of how to rebuild an index in a production system without losing the ability to utilize the index:

CREATE INDEX CONCURRENTLY new_index_name ON table_name (column_name);
DROP INDEX CONCURRENTLY old_index_name;
ALTER INDEX new_index_name RENAME TO old_index_name;

Note that using this method means you will briefly have two copies of the same index, so if disk space is an issue you may need to take other approaches.

Users who have skipped one or more update releases may need to run additional, post-update steps; please see the release notes for earlier versions for details.

Links:

Categories: Database, Open Source

Google Summer of Code 2016 wrap-up: LabLua

Google Open Source Blog - Wed, 02/08/2017 - 18:00
This is the final guest post from the students, mentors and organization administrators that participated in Google Summer of Code (GSoC) 2016. We’ve seen recaps of student work and lessons learned, which you can check out the rest of the series as we gear up for this year’s program.


LabLua is a lab at PUC-Rio dedicated to research on programming languages, with emphasis on the Lua language. Lua is a powerful, fast, lightweight, embeddable scripting language that is used in many industrial applications, and on many embedded systems and games.
We were very happy to participate in Google Summer of Code (GSoC) for the third time, and to mentor eight fine students that all completed their projects successfully. We thank them, and Google, for this extraordinary contribution to our research and development work.
Here is a brief summary of this year's projects:
Next Generation of the LuaRocks test suite - Robert KarasekLuaRocks is the package manager for Lua modules. Its test suite was implemented as a big shell script that performed only black-box testing and ran only on Linux. The goal for this project was to port the test suite to Lua, improving its portability and allowing more types of tests so we could improve test coverage.
Robert ported the test suite to Lua using Busted. His new test suite, now merged into LuaRocks, runs on Linux and Mac OS X, accessible via Travis CI, as well as Windows, accessible via AppVeyor. 
This was a welcome addition, bringing greater confidence to developers. Robert improved the checks in existing tests and wrote many new ones, including a new mock-server for testing a client API for uploading packages to the repository.
Typed Lua Typechecker - Tomasz Dyczek 
Typed Lua provides static type checking for the Lua language. Typed Lua extends the syntax of Lua 5.3 to introduce type annotations, and performs local type inference for more precise detection of unannotated expressions.
Tomasz implemented the core of Typed Lua in Haskell. Tomasz's implementation parses code written in a syntax close to the abstract syntax of Typed Lua, then type checks the generated AST. Besides providing a support for testing and reasoning about new features, Tomasz's typechecker can be also used to validate tests to be included in Typed Lua's test suite.
Classes and Generics for Typed Lua - Kevin Clancy
Kevin worked on the implementation of a class system for Typed Lua. He also added parametric polymorphism (generics) for classes and existing Typed Lua types, such as functions and tables.
Kevin's work currently lives in its own branch, but will be merged into the main branch soon. Meanwhile, Kevin has written a detailed post explaining all the features he implemented
Improving Error Reporting in PEG Parsers - Matthew Allen Go 
LPegLabel is an extension of LPeg, a pattern matching tool for Lua, based on Parsing Expression Grammars (PEGs). LPegLabel supports labeled failures, a facility that improves error reporting and recovery for PEG-based parsers.
The goal of this project was to use LPegLabel to write parsers with good error reporting. These parsers could then be used by the Lua community and also serve as a guide for LPegLabel users. Because LPegLabel is a young tool, another important contribution was to improve the tool's usability.
Matthew achieved both goals. He developed a parser for Lua 5.3, which has been incorporated into the new release of lua-parser (1.0.0), and improved LPegLabel’s usability with work on its API and documentation.
Improving elasticsearch-lua tests and build - Dhaval Kapil
Elasticsearch is a distributed and scalable search engine written in Java that offers a REST API accessed through JSON. During GSoC 2015, Dhaval implemented elasticsearch-lua, a client for the Lua language following a model similar to clients written in Python and PHP.
During GSoC 2016, Dhaval worked on improving elasticsearch-lua. He added a test suite, documented the entire codebase, and updated the current client to work with the newest version of Elasticsearch.
Dhaval went above and beyond, creating a new library called luaver. This work was motivated by having to frequently switch between different versions of Lua while developing the test suite. A full blog post about his project can be found here.
Admin Center and Elasticsearch integration for Sailor - Nikhil Ramesh 
Sailor is a web framework with a model-view-controller (MVC) architecture. Like other web frameworks, such as Ruby on Rails and Django, it is designed to make development faster by making some assumptions and conventions and encouraging principles like Don’t Repeat Yourself (DRY).
Nikhil focused on extending Sailor. The first feature he worked on was an Admin Center, which is a web interface for configuring an application. He also integrated Sailor and elasticsearch-lua, allowing Elasticsearch indexes to be stored as Sailor Models. His work is currently pending as a pull request and will soon be merged.
Extending the online tutorial of Céu with Emscripten and SDL - Margarit Vicentiu
Céu is a language for developing reactive applications such as video games and embedded systems. Its compiler generates output in plain C to integrate easily with the underlying platform (e.g. Arduino, SDL). For this project, we wanted to integrate Céu with Emscripten in order to run applications in a web browser.
Vicentiu started with Céu’s online tutorial, which is a server-side application: the user types code in a text area and hits the send button; the server receives the code, executes it, and sends the output back to the user. During the summer, Vicentiu made most of the examples compile with Emscripten and run in real-time on the user’s screen.
Our next goal is to make the graphical examples with user interactions also work in the browser, and Vicentiu plans to continue contributing to the project to achieve this goal.
An automatic generator of WSDL documents for LuaSOAP - Victor Dias
LuaSOAP is a library for working with the Simple Object Access Protocol (SOAP). WSDL is an XML format for describing network services; it is used to describe operations, messages and types offered by Web Services.
This summer Victor extended LuaSOAP's WSDL support by building a software layer for the automatic generation of WSDL documents. This new layer eases the description of most WSDL "bureaucracy" -- types, operations, ports, messages -- which have no counterparts in Lua. He also improved the test suite and the documentation. Victor's work will be integrated into the next version of LuaSOAP.
By Ana Lúcia de Moura, Organization Administrator for LabLua
Categories: Open Source