Skip to content

Software Development News: .NET, Java, PHP, Ruby, Agile, Databases, SOA, JavaScript, Open Source

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Using Docker to develop with Couchbase

NorthScale Blog - 16 hours 24 min ago

Rafael Ugolini is a full stack software developer currently based in Brussels, Belgium. He has been working with software development for more than 10 years and is lately focused on designing web solutions and developing using Python and JavaScript. Rafael Ugolini is Senior Software Developer at Famoco.

FullSizeRender Introduction

Docker is a great project that is helping developers around the world run applications in containers. This not only helps shipping software faster, but it also results with the famous ‚Äúit works in my machine‚ÄĚ phrase. In this article I will explain how to create a modular Couchbase image that doesn‚Äôt require any Web UI interaction to have a ready-to-go database for you.

All the code is available online here.

Dockerfile

The first step is to create the Dockerfile.

Couchbase Version

FROM couchbase/server:enterprise-4.6.1

This example is based on Couchbase Server Enterprise 4.6.1, but you can feel free to change to the specific version you are running in your environment.

Memory Configuration

ENV MEMORY_QUOTA 256
ENV INDEX_MEMORY_QUOTA 256
ENV FTS_MEMORY_QUOTA 256

All the values here are in MB:

– MEMORY_QUOTA: per node data service ram quota

– INDEX_MEMORY_QUOTA: per node index service ram quota

– FTS_MEMORY_QUOTA: per node index service ram quota

Services

ENV SERVICES "kv,n1ql,index,fts"

These are the services that will be available for the node created:

– kv: Data

– n1ql: Query

– index: Index

– fts: Full-Text Search

Credentials

ENV USERNAME "Administrator"
ENV PASSWORD "password"

Username and password to be used in Couchbase Server.

Cluster Options

ENV CLUSTER_HOST ""
ENV CLUSTER_REBALANCE ""

These options are only used if you want to add more than one node in the cluster.

– CLUSTER_HOST: hostname of the cluster for this node to join

– CLUSTER_REBALANCE: set “true” if you want the cluster to rebalance after the node is joined

Entrypoint

COPY entrypoint.sh /config-entrypoint.sh

ENTRYPOINT ["/config-entrypoint.sh"]

The Couchbase Server image already ships with an entrypoint.sh script and we don’t want to override it. The trick here is to copy our version of entrypoint.sh to /config-entrypoint.sh,¬†run Couchbase Server entrypoint.sh in the background,¬†and after configuring the node attach the script back to the original ENTRYPOINT.

Entrypoint

The ENTRYPOINT is used in combination with the original script from the Couchbase Server image. Let’s go line by line to understand how it works.

Initialize Couchbase Server

# Monitor mode (used to attach into couchbase entrypoint)
set -m
# Send it to background
/entrypoint.sh couchbase-server &

First we use set -m to enable job control, process running in background (like the original ENTRYPOINT) run in a separate process group. This option is turned off by default in non-interactive mode, like scripts.

Util Functions

# Check if couchbase server is up
check_db() {
 curl --silent http://127.0.0.1:8091/pools > /dev/null
 echo $?
}

This function is used to check when Couchbase Server starts answering HTTP calls.

# Variable used in echo
i=1
# Echo with
numbered_echo() {
echo "[$i] $@"
i=`expr $i + 1`
}

This is just a util function, add a number before any echo in the script to count the steps taken automatically.

# Parse JSON and get nodes from the cluster
read_nodes() {
cmd="import sys,json;"
cmd="${cmd} print(','.join([node['otpNode']"
cmd="${cmd} for node in json.load(sys.stdin)['nodes']"
cmd="${cmd} ]))"
python -c "${cmd}"
}

In order to parse the output of the nodes in Couchbase Server API, I’m using a function which runs ython to read STDIN, transform it to JSON and the Couchbase nodes. This is used for rebalancing.

Configure the Node

# Wait until it's ready
until [[ $(check_db) = 0 ]]; do
>&2 numbered_echo "Waiting for Couchbase Server to be available"
sleep 1
done

echo "# Couchbase Server Online"
echo "# Starting setup process"

The first step is to wait until the server is ready, then using the function numbered_echo you can see how long it took for Couchbase Server to have the API calls available.

HOSTNAME=`hostname -f`

# Reset steps
i=1

Then we set a variable HOSTNAME to be used in all the API calls we do and we also reset the counter from numbered_echo by setting it to 1.

numbered_echo "Initialize the node"
curl --silent "http://${HOSTNAME}:8091/nodes/self/controller/settings" \
-d path="/opt/couchbase/var/lib/couchbase/data" \
-d index_path="/opt/couchbase/var/lib/couchbase/data"

numbered_echo "Setting hostname"
curl --silent "http://${HOSTNAME}:8091/node/controller/rename" \
-d hostname=${HOSTNAME}

 

First thing to do is to set up disk storage configuration and then we set the hostname.

Joining a Cluster

if [[ ${CLUSTER_HOST} ]];then
numbered_echo "Joining cluster ${CLUSTER_HOST}"
curl --silent -u ${USERNAME}:${PASSWORD} \
"http://${CLUSTER_HOST}:8091/controller/addNode" \
-d hostname="${HOSTNAME}" \
-d user="${USERNAME}" \
-d password="${PASSWORD}" \
-d services="${SERVICES}" > /dev/null

If CLUSTER_HOST is set, the script will try to add the current container to the cluster.

if [[ ${CLUSTER_REBALANCE} ]]; then
# "Unexpected server error without the sleep 2
sleep 2
numbered_echo "Retrieving nodes"
known_nodes=$(
curl --silent -u ${USERNAME}:${PASSWORD} http://${CLUSTER_HOST}:8091/pools/default | read_nodes
)

numbered_echo "Rebalancing cluster"
curl -u ${USERNAME}:${PASSWORD} \
"http://${CLUSTER_HOST}:8091/controller/rebalance" \
-d knownNodes="${known_nodes}"
fi

else

After adding the node into the cluster, the script can also check for the variable CLUSTER_REBALANCE to see if it needs to rebalance the cluster automatically. This is where we use the Python function to read the nodes from /pools/default endpoint.

Not joining a cluster

numbered_echo "Setting up memory"
curl --silent "http://${HOSTNAME}:8091/pools/default" \
-d memoryQuota=${MEMORY_QUOTA} \
-d indexMemoryQuota=${INDEX_MEMORY_QUOTA} \
-d ftsMemoryQuota=${FTS_MEMORY_QUOTA}

Memory settings for the services.

numbered_echo "Setting up services"
curl --silent "http://${HOSTNAME}:8091/node/controller/setupServices" \
-d services="${SERVICES}"

Services to be used by the node.

numbered_echo "Setting up user credentials"
curl --silent "http://${HOSTNAME}:8091/settings/web" \
-d port=8091 \
-d username=${USERNAME} \
-d password=${PASSWORD} > /dev/null

fi

Set up the credentials for the node.

Finalize

# Attach to couchbase entrypoint
numbered_echo "Attaching to couchbase-server entrypoint"
fg 1

To end the script, we attach it to the original ENTRYPOINT.

Example

To demonstrate how to use it, I will be using the image registered in Docker Hub with the code here.

Single node

 

docker run -ti --name couchbase-server-nosetup \
-h node1.cluster \
-p 8091-8093:8091-8093 \
-p 11210:11210 \
-p 4369:4369 \
-p 21100-21299:21100-21299 \
rugolini/couchbase-server-nosetup

This runs a single node using the minimum required memory and the default credentials (Administrator/password) registered in the image. All the network ports Couchbase Server uses are exposed as well.

docker run -ti --name couchbase-server-nosetup \
-h node1.cluster \
-p 8091-8093:8091-8093 \
-p 11210:11210 \
-p 4369:4369 \
-e MEMORY_QUOTA=512 \
-e INDEX_MEMORY_QUOTA=512 \
-e FTS_MEMORY_QUOTA=512 \
-e USERNAME=admin \
-e PASSWORD=adminadmin \
-p 21100-21299:21100-21299 \
rugolini/couchbase-server-nosetup

The command above plays a little with the environment variables available in the Dockerfile.

Cluster

In this example, we will connect 3 nodes in the cluster.

docker network create couchbase

 

We must first create a network Couchbase where we will connect all nodes.

docker run -ti --name node1.cluster \
-p 8091-8093:8091-8093 \
-p 11210:11210 \
-p 4369:4369 \
-p 21100-21299:21100-21299 \
-h node1.cluster \
--network=couchbase \
rugolini/couchbase-server-nosetup

Then we create the first node.

docker run -ti --name node2.cluster \
--network=couchbase \
-h node2.cluster \
-e CLUSTER_HOST=node1.cluster \
-e CLUSTER_REBALANCE=true \
rugolini/couchbase-server-nosetup

 

Since all the network ports are exposed in the first node, it’s not necessary to expose them here.

Attention to the detail that CLUSTER_HOST is set to node1.cluster which is the hostname of the first node and CLUSTER_REBALANCE is also set to true. Once the node is added to the cluster, it will rebalance automatically.

docker run -ti --name node3.cluster \
--network=couchbase \
-h node3.cluster \
-e CLUSTER_HOST=node1.cluster \
rugolini/couchbase-server-nosetup

 

The node3 is also added to the cluster, but since CLUSTER_REBALANCE wasn’t set, it will require manual rebalance of the cluster for it to become available.

 

This post is part of the Couchbase Community Writing Program

The post Using Docker to develop with Couchbase appeared first on The Couchbase Blog.

Categories: Architecture, Database

Engaging for growth: Introducing the industry’s first Engagement Database

NorthScale Blog - Tue, 05/23/2017 - 13:55

Today at Couchbase, we tread a new path by carving out a new category of database: the Engagement Database.

What is an Engagement Database, you ask me ‚Äď a guy steeped in the technology side of the business? It‚Äôs something that addresses a need we discovered when speaking with our customers over the past year. We asked them, ‚ÄúWhat kind of applications are you building?‚ÄĚ It became obvious that just being ‚ÄúNoSQL‚ÄĚ didn‚Äôt address the challenges our customers were facing.

The applications our customers are focused on, as part of their organization’s‚Äô overall digital innovation initiatives, really comes down to enhancing the customer experience. With consumers‚Äô diminishing attention spans and increasing fickleness when it comes to final purchase decisions, customer experience is a fierce battleground on which today‚Äôs businesses are fighting.

With more and more businesses seeing customer experience equating to competitive advantage, new technologies continue to evolve to unleash data‚Äôs potential and create those amazing experiences. However, it‚Äôs a simple fact that databases ‚Äď from transactional databases to the majority of NoSQL solutions ‚Äď have not been optimized to support and nurture the interactions customers desire. And scale only makes the problem worse.

So far, attempts to solve this problem have led to organizations using a variety of different databases for specific use cases ‚Äď leading to a difficult-to-manage ‚Äėdatabase sprawl.‚Äô The solution is a comprehensive ‚Äėengagement database,‚Äô or one that allows enterprises to strategically and expeditiously serve up data when designing customer interactions and experiences ‚Äď especially when they need to perform at scale across all channels and devices. This is the Couchbase Data Platform.

The Couchbase Data Platform was designed to harness the full potential of dynamic data, at any scale, across any channel or device. Built on the most powerful NoSQL technology, the Couchbase Data Platform ‚Äď as the industry‚Äôs first Engagement Database ‚Äď makes it simple to continually reinvent the customer experience. No other database provides the capabilities that are required to create these experiences from a single platform.

Today we also unveiled innovation behind the industry‚Äôs first Engagement Database: the latest Beta release of Couchbase Server 5.0 and preview of Couchbase Mobile 2.0, both part of the Couchbase Data Platform. ¬†The enhanced product suite provides improved developer agility, query performance, and easier cluster management, all of which enable enterprises to build amazing customer engagement applications. Finally, the jewel in the crown is the fully integrated full text search capability–not just for server deployments in the cloud, but also for mobile devices on the edge. To summarize, key enhancements include:

Couchbase Server 5.0 Beta
  • Richer customer experiences with built-in full text search with independent scaling all in the same Couchbase cluster ¬†
  • More responsive applications and efficient data management, with in-memory data sets (no disk needed) including support for query, indexing and high availability replication for in-memory data.
  • Simpler application development that leverages enhanced N1QL query monitoring and debugging, and technology leading adaptive indexing, and built-in multi data center programmability. ¬†
  • Enhanced security options, including fine-grained role based access control (RBAC) support for users and applications.
  • More available applications, from fast node failover in seconds and automatic index replication.
  • Strong ecosystem improvements for ETL, and for public and hybrid cloud solutions
Couchbase Mobile 2.0 Preview
  • Richer customer experiences by providing built-in full text search for mobile applications.
  • Simplified application development with support for N1QL-like API, as well as a new API that provides built-in domain data modeling support
  • Agile development of collaborative, multi-user apps with automatic conflict management on the edge.
  • Faster apps with delta synchronization for more efficient data management from the edge to the cloud

We are very excited to share this news in front of hundreds of customers and partners at Couchbase Connect in New York this week. Stay tuned as we share additional feedback and reaction from attendees and industry pundits next week.

The post Engaging for growth: Introducing the industry’s first Engagement Database appeared first on The Couchbase Blog.

Categories: Architecture, Database

Announcing Couchbase Server 5.0 Beta

NorthScale Blog - Mon, 05/22/2017 - 23:21

We’re excited to pull back the curtain on the beta release of Couchbase Server 5.0. With this release, Couchbase provides the world’s first Engagement Database, built on the most powerful NoSQL technology. This platform delivers unparalleled performance at any scale, while providing unmatched agility and manageability.

To meet the requirements of an Engagement Database, this beta release comes with new key capabilities that further strengthen our core characteristics of agility, performance, and manageability.

  1. Full-text search
  2. Powerful indexing & querying
  3. Secure platform
  4. Performance, scalability, and high-availability
  5. Enhanced management, developer tools
  6. Big data connectors

With these additions and much more, Couchbase Server 5.0 Beta is a huge release that packs a massive punch for enterprises looking to transform their digital business.

Download Couchbase Server 5.0 Beta¬† What’s new in 5.0 Beta?

Let’s dive right in!

Full-text Search – built right in

Need to enable flexible and powerful Search capabilities?¬†No digital application is complete without search. With 5.0, Couchbase search is becoming an integral part of the scalable data platform. Full-text search (FTS) provides the ability to index Couchbase documents and query them rapidly using a variety of indexing methods, text analyzers, and languages ‚Äď without leaving the Couchbase Data Platform and without duplicating any data. The search index gets in-memory updates as the data changes. To learn more about FTS, see Full-text search reference.

Secure Platform

For digital applications, being compliant is not just nice-to-have, but a must-have. Built on the role-based access control (RBAC) security model from 4.5, RBAC for applications in 5.0 Beta allows you to segregate access and closely align user roles in Couchbase with the actual roles they hold within your organization. This allows your application services and users to access the information they need – nothing more, nothing less.

The bottom line is that with RBAC for applications you are now closer to meeting your security compliance requirements. To learn more, read about RBAC in our documentation.

Powerful indexing, querying, and search

Couchbase Server 5.0 Beta adds a unique new way to provide ad hoc search queries through N1QL. Adaptive indexes can efficiently look up any of the indexed fields ‚Äď without requiring you to create multiple composite indexes or different index key combinations! For more information, read about adaptive indexes in our documentation.

CREATE INDEX `aidx_self` ON `travel-sample`((distinct (PAIRS(self))));

EXPLAIN SELECT * FROM `travel-sample` USE INDEX(aidx_self) WHERE (city LIKE "San%") and type = 'airport';

Want to identify a long-running query and tune your query performance? With Couchbase Server 5.0, you get a detailed visual query plan with execution timing and other query monitoring statistics that provide detailed insights into the query execution process. Check our documentation to learn how you can monitor your N1QL queries.

Want to join external data with the data stored in Couchbase? Whether it’s social data, map data, or any other JSON data on the web, now there’s a way with N1QL CURL to interact with it, and integrate to build powerful applications. Check our documentation to learn more about N1QL CURL.

Performance, scalability, and high availability

From the very beginning, performance has been one of the key reasons why enterprises have chosen Couchbase for mission-critical applications. With 5.0 Beta we have continued to push the boundaries even further to provide you with unparalleled performance at any scale!

Want better performance at a much lower cost? Now, with the new ephemeral buckets feature in Couchbase 5.0 Beta, you can reduce the total cost of ownership (TCO) by eliminating the disk component for your Couchbase buckets, and get highly consistent in-memory performance without disk-based fluctuations. For more information, see here.

For a mission-critical Engagement Database, robust failure detection and recovery are key. The new fast-failover feature in Couchbase 5.0 Beta provides a robust failure detection mechanism to reduce the time window of a failure detection from 30 seconds to less than 10 seconds. This means, increased 9’s of availability for your application. For more information, see here.

With Couchbase 5.0 Beta, you can not only create indexes to speed up and scale queries to a new level, but also enjoy better index availability and manageability. Just specify the number of index replicas to create, and the system will dynamically manage the placements of the index replicas across different nodes, server groups, and availability zones. Couchbase Server 5.0 also brings support for rebalancing indexes without any system downtime. For more information, see here.

5.0 Beta also adds several other N1QL performance enhancements. Some of the notable ones are:

  1. Indexing and querying on meta() fields
  2. Complex predicate pushdowns
  3. Pagination pushdowns 
  4. Operator pushdown
Enhanced management & developer tools

When you launch Couchbase Server 5.0 Beta, you will notice a modern web console interface that is engineered for intelligence, comfort, and speed. This redesigned interface offers a new look and streamlines your common tasks and workflows.

 

Find it hard to read JSON or text-based N1QL query plans? 5.0 Beta has the answer ‚Äď he visual query plan feature has arrived!

The visual query plan feature provides a graphical representation of the query execution process as data flows visually from one query operator to another, highlighting the most expensive operations.

You can also use query monitoring to see currently active queries and how long they’ve been running, to view the longest running completed queries, and see statistics on the execution of prepared queries.


For building applications with Couchbase Server 5.0 Beta as fast as possible we have updated our SDKs to support many of the new critical features you‚Äôve seen above ‚Äď including RBAC, fast failover, ephemeral buckets, and full-text search. We‚Äôve also introduced many other language-specific improvements, as well as improved integration with Spring Data and .NET Core support. For more information, check out the release notes of each language. ¬†

Couchbase 5.0 Beta also provides integrations with Spark 2.1, and a developer preview connector for Talend Open Studio 6.3.1.

Dig deeper!

We have just scratched the surface of Couchbase Server 5.0 with the capabilities mentioned above and there are many more. Our new 5.0 documentation will help you to dig deeper.

OK! How do I get my hands on 5.0 beta?

Remember this before you take the plunge:

This is a beta version of Couchbase Server 5.0 and is intended for development purposes only.

To use 5.0 Beta, you need to do a fresh install. This release is not supported in production.

We consider beta releases to have some rough edges and bugs.

Overall, the release is still under active development, which means that
you can have a big impact on the final version of the product by providing feedback and observations.

It is easy to get your hands on the beta:

  1. Download Couchbase Server 5.0 Beta from our download page.
  2. Looking to develop in Java, .Net, Node.js, PHP, or other languages with native SDKs? Download the latest SDK version under the ‚Äúclient libraries‚ÄĚ section on the downloads page.
  3. Don’t forget to check out our 5.0 Beta release notes.

Finally, good luck building your applications with Couchbase Server 5.0 Beta, and we look forward to your valuable feedback.

The post Announcing Couchbase Server 5.0 Beta appeared first on The Couchbase Blog.

Categories: Architecture, Database

Announcing Couchbase Mobile 2.0 Developer Preview

NorthScale Blog - Mon, 05/22/2017 - 23:16

Couchbase Mobile 2.0 is a groundbreaking new release for Couchbase Mobile. We’ve reimagined the developer experience with a cross-platform common core, new simplified API, and automated conflict resolution that can be customized. In this release, we are bringing N1QL queries and full-text search capabilities to mobile. Read on to learn all about the most advanced NoSQL mobile database on the planet!

Couchbase Lite

We have rewritten Couchbase Lite in the 2.0 version. The database core engine, internally referred to as ‚ÄúCouchbase Lite core,‚ÄĚ has been implemented in C/C++. With a common core, the size of the codebase has been significantly reduced resulting in better manageability. It also allows for easy porting to low fidelity devices that enable new IoT use cases and opens new markets for Couchbase mobile developers.

There are language-specific bindings on top of the common core for iOS, .NET, and Java. During development, we’ve seen a 6x improvement in performance between 1.x and 2.0.

New simplified API

With built-in thread safety, mutable properties, typed accessors and blobs for accessing attachments, you will find the reimagined API easy to learn. We’ve gone through multiple iterations of the API as we engaged with our developer community early on. Your feedback has been invaluable in shaping the current revision of the APIs and we look forward to your continued support.

Fluent API for N1QL queriesDatabase queries have changed significantly in this release. Couchbase Lite 2.0 uses semantics based on the N1QL query language defined by Couchbase. The query API has two styles, builder and fluent, and the implementation chosen would depend on the development platform.

The API allows you to access multiple Couchbase Lite databases with cross-database joins. This API will be familiar if you’ve used Core Data, or other query APIs based on SQL (like jOOQ).

Full-text search

In this release, users can perform full-text searches on the JSON documents stored in Couchbase Lite. You can now bring to your mobile applications what Google, Yahoo, and Bing do with HTML on the web. The API for using full-text search is not very different from the query API ‚Äď users can search for text, text fragments, and text connected by binary operators, and Couchbase Lite finds the set of JSON documents that best match those terms.

Replication

Couchbase Mobile 2.0 uses a new replication protocol, based on WebSockets. This protocol has been designed to be fast, efficient, easier to implement, and symmetrical between client/server. Even though the replication protocol has changed, Couchbase Lite 1.x clients will be able to work with Couchbase Mobile 2.0 deployments since Couchbase Sync Gateway continues to support both clients.

However, the new replication protocol is incompatible with version 1.x, and with CouchDB-based databases including PouchDB and Cloudant.

But the new replicator is faster than the old one ‚Äď we’ve seen up to twice the speed on iOS devices, and even greater improvement on Android.

Automated conflict management

We’ve taken a completely different approach to conflict management. An application is no longer required to handle conflicts out of band and keep track of various conflicting revisions. Couchbase Lite 2.0 will detect a conflict while saving a document or during replication and invoke an app-defined conflict resolver. The conflict resolution is designed for flexibility that will allow developers to tailor it for their specific needs.

To get started  

The post Announcing Couchbase Mobile 2.0 Developer Preview appeared first on The Couchbase Blog.

Categories: Architecture, Database

Synchronizing Images Between Android and iOS with NativeScript, Angular, and Couchbase

NorthScale Blog - Thu, 05/18/2017 - 15:00

A few weeks ago I had written a guide that demonstrated saving captured images to Couchbase Lite as base64 encoded string data in a NativeScript with Angular mobile application. While the previous guide worked for both Android and iOS, the data was localized to the device. What if you wanted to synchronize the images between devices or even store them in the cloud?

We’re going to see how to use Couchbase Mobile to synchronize image data between devices and platforms in a NativeScript with Angular application.

Going forward you should note that this is part two in the series. ¬†This means that if you haven’t already followed the previous tutorial and gotten a working version of the project, you should put this tutorial on hold. Start with the guide,¬†Save Captured Images in a NativeScript Angular Application to Couchbase, then work your way into¬†synchronizing the images.

NativeScript Couchbase Photos

The above animated image will give you a rough idea of what we’re after. ¬†We want to be able to synchronize the saved images between Android and iOS using Sync Gateway and optionally Couchbase Server.

The Requirements

The prerequisites for this guide are similar to what was found in the previous. ¬†You’ll need the following:

  • NativeScript CLI
  • Android SDK for Android or Xcode for iOS
  • Couchbase Sync Gateway
  • Couchbase Server (optional)

You’ll notice that Sync Gateway and optionally Couchbase Server are the new requirements of this guide in the series. ¬†We’ll need these for synchronization to actually happen. ¬†If you’re unfamiliar, Sync Gateway is the synchronization middleware and Couchbase Server is the remote database server.

Configuring Sync Gateway for Replication

To use Sync Gateway we’ll need to define a configuration as to how synchronization happens and things like that.

Create a sync-gateway-config.json file somewhere on your computer that contains the following information:

{
    "log":["CRUD+", "REST+", "Changes+", "Attach+"],
    "databases": {
        "image-database": {
            "server":"walrus:data",
            "sync":`
                function (doc) {
                    channel (doc.channels);
                }
            `,
            "users": {
                "GUEST": {
                    "disabled": false,
                    "admin_channels": ["*"]
                }
            }
        }
    }
}

In the above configuration file we are saving everything to¬†walrus:data which is an in-memory solution rather than persisting it to Couchbase Server. ¬†The remote database is called¬†image-database, but it doesn’t have to match what we have in our mobile application code.

For simplicity everyone will be able to read and write data in the same channel as a guest.

To run Sync Gateway, execute the following:

/path/to/sync_gateway /path/to/sync-gateway-config.json

You should be able to access Sync Gateway from your web browser at http://localhost:4984/_admin/ and view everything that is being synchronized, often referred to as replicated.

Adding the Logic for Synchronizing Image Data

The actual code involved towards getting replication working in our NativeScript with Angular application is minimal.

Open the project’s¬†app/app.component.ts file and include the following TypeScript code:

import { Component, OnInit, NgZone } from "@angular/core";
import { Couchbase } from "nativescript-couchbase";
import * as Camera from "camera";
import * as ImageSource from "image-source";

@Component({
    selector: "ns-app",
    templateUrl: "app.component.html",
})
export class AppComponent implements OnInit {

    private database: any;
    private pushReplicator: any;
    private pullReplicator: any;
    public images: Array<any>;

    public constructor(private zone: NgZone) {
        this.database = new Couchbase("image-database");
        this.database.createView("images", "1", function(document, emitter) {
            if(document.type && document.type == "image") {
                emitter.emit(document._id, document);
            }
        });
        this.pushReplicator = this.database.createPushReplication("http://192.168.57.1:4984/image-database");
        this.pullReplicator = this.database.createPullReplication("http://192.168.57.1:4984/image-database");
        this.pushReplicator.setContinuous(true);
        this.pullReplicator.setContinuous(true);
        this.images = [];
    }

    public ngOnInit() {
        this.pushReplicator.start();
        this.pullReplicator.start();
        this.database.addDatabaseChangeListener(changes => {
            for(let i = 0; i < changes.length; i++) {
                this.zone.run(() => {
                    let image = ImageSource.fromBase64(this.database.getDocument(changes[i].getDocumentId()).image);
                    this.images.push(image);
                });
            }
        });
        let rows = this.database.executeQuery("images");
        for(let i = 0; i < rows.length; i++) {
            this.images.push(ImageSource.fromBase64(rows[i].image));
        }
    }

    public capture() {
        Camera.takePicture({ width: 300, height: 300, keepAspectRatio: true, saveToGallery: false }).then(picture => {
            let base64 = picture.toBase64String("png", 70);
            this.database.createDocument({
                "type": "image",
                "image": base64,
                "timestamp": (new Date()).getTime()
            });
        }, error => {
            console.dump(error);
        });
    }

}

The above code includes everything from the first part of the series as well as this part of the series. ¬†We’re going to break down only what has been added in regards to replication.

In the constructor method we define where we are going to push data to and where we are going to pull data from.

this.pushReplicator = this.database.createPushReplication("http://192.168.57.1:4984/image-database");
this.pullReplicator = this.database.createPullReplication("http://192.168.57.1:4984/image-database");
this.pushReplicator.setContinuous(true);
this.pullReplicator.setContinuous(true);

This is to be done continuously for as long as the application is open.

Make sure you use the correct host or IP address for Sync Gateway. ¬†If you’re using Genymotion like I am, localhost will not work. ¬†You’ll have to figure out the correct IP addresses.

In the ngOnInit method we start the replication process and configure a listener.

this.pushReplicator.start();
this.pullReplicator.start();
this.database.addDatabaseChangeListener(changes => {
    for(let i = 0; i < changes.length; i++) {
        this.zone.run(() => {
            let image = ImageSource.fromBase64(this.database.getDocument(changes[i].getDocumentId()).image);
            this.images.push(image);
        });
    }
});

Any time there is a change in the database we will loop through them and load the base64 data.  This example is simple so there are no updates or deletions of images.  If this was the case, our listener would be a little more complex in logic.

The reason we are using an Angular NgZone is because the listener operates on a different thread.  By zoning, we can take the data and make sure the UI updates correctly.

That’s all we had to do to get the images synchronizing between device and server. ¬†Easy right?

Conclusion

You just saw how to synchronize image data between devices and platforms using NativeScript, Angular, and Couchbase. This was a followup to the previous tutorial I wrote called, Save Captured Images in a NativeScript Angular Application to Couchbase, where we got the initial application up and running.

In case you’d rather not store your images in the database, you might consider creating an API that uses an object storage like Minio or Amazon S3. ¬†I’ve written a tutorial on creating an API that saves to Minio that might help.

For more information on using Couchbase with Android and iOS, check out the Couchbase Developer Portal.

The post Synchronizing Images Between Android and iOS with NativeScript, Angular, and Couchbase appeared first on The Couchbase Blog.

Categories: Architecture, Database

NDP Episode #18: Microsoft DocumentDB for NoSQL in the Cloud

NorthScale Blog - Tue, 05/16/2017 - 16:00

I am pleased to announce that the latest episode of the NoSQL Database Podcast titled,¬†Microsoft DocumentDB for NoSQL in the Cloud, has been published to all the major podcasting networks. ¬†In this episode I am joined by¬†Kirill Gavrylyuk from Microsoft’s Azure team where we talk about the NoSQL database, DocumentDB, now known as Azure Cosmos DB.

This episode can be found for free on iTunes, Pocket Casts, and various other networks, but it can also be found below.

http://traffic.libsyn.com/nosql/TNDP_-_Episode_18_-_Microsoft_DocumentDB_for_NoSQL_in_the_Cloud.mp3

If you want to learn more about DocumentDB, I encourage you to follow Kirill on Twitter. ¬†If you’re interested in more information about the podcast or want to make suggestions, contact me on Twitter at @nraboy.

Want to learn about the NoSQL database, Couchbase?  Check out the Couchbase Developer Portal for documentation and examples.

The post NDP Episode #18: Microsoft DocumentDB for NoSQL in the Cloud appeared first on The Couchbase Blog.

Categories: Architecture, Database

Data Modeling NoSQL Documents in MongoDB vs Couchbase Server

NorthScale Blog - Thu, 05/11/2017 - 16:00

When I’m out at events, I get a lot of questions regarding the differences between MongoDB and Couchbase Server as they are both in the NoSQL space and are both document databases. One particular question is related to data modeling. MongoDB uses BSON and Couchbase uses JSON, so wouldn’t the data model be different?

We’re going to take a look at some MongoDB document models and see how to accomplish the same in Couchbase with minimal to no effort.

Let me start by saying that both MongoDB and Couchbase have some great documentation on modeling NoSQL documents. Both refer to to a similar set of practices that we’ll explore.

Embedded Data Models

Given that you’re able to have incredibly complex NoSQL documents with nested objects and nested arrays, we can do things a little differently than in a relational database.

Take the following MongoDB document for example:

{
    "_id": <ObjectId>,
    "first_name": "Nic",
    "last_name": "Raboy",
    "address": {
        "city": "Mountain View",
        "state": "California"
    },
    "social_media": [
        {
            "type": "twitter",
            "url": "https://www.twitter.com/nraboy"
        },
        {
            "type": "mastodon",
            "url": "https://toot.cafe/@nraboy"
        }
    ]
}

In the above example we have nested objects and arrays embedded into a single document that is defined by an object id in some MongoDB Collection.

In Couchbase, the same MongoDB document might look something like this:

{
    "_type": "person",
    "first_name": "Nic",
    "last_name": "Raboy",
    "address": {
        "city": "Mountain View",
        "state": "California"
    },
    "social_media": [
        {
            "type": "twitter",
            "url": "https://www.twitter.com/nraboy"
        },
        {
            "type": "mastodon",
            "url": "https://toot.cafe/@nraboy"
        }
    ]
}

You’ll probably notice that the above Couchbase example is nearly identical to the MongoDB example with the exception of the¬†_id property becoming a¬†_type property. ¬†In Couchbase, the document id exists as a meta property rather than within the document itself. ¬†Since there is no concept of Collections in Couchbase, documents are usually differentiated by a type property, but it is not a requirement.

Embedding objects and arrays into single document is great because it limits the amount of steps required to do common operations on the data. However, as the document becomes larger, performance could become an issue.

This is where the next document modeling approach comes into play.

Normalized or Referred Data Models

Anyone coming from a relational database knows that data should be normalized across multiple tables within the database. These tables are then joined in some fashion when the data needs to be accessed together.

The concept of this can still be applied in NoSQL, even if it is not exactly the same as you’d find in a relational database.

Going back to the first example, let’s do what MongoDB refers to as a normalized data model:

{
    "_id": <ObjectId1>,
    "first_name": "Nic",
    "last_name": "Raboy",
    "address": <ObjectId2>,
    "social_media": [
        {
            "type": "twitter",
            "url": "https://www.twitter.com/nraboy"
        },
        {
            "type": "mastodon",
            "url": "https://toot.cafe/@nraboy"
        }
    ]
}

The embedded model changed slightly by moving address into its own document.  In this case the address property now equals an object id and the new document looks like this:

{
    "_id": <ObjectId2>,
    "city": "Mountain View",
    "state": "California"
}

Splitting the document into multiple documents could help in several areas. The documents are now smaller and operations on them can be potentially faster.  The data is also now normalized in the sense that now multiple people can exist at the same address without having to worry about data duplication that would exist in the embedded model.

Per the MongoDB documentation:

… However, client-side applications must issue follow-up queries to resolve the references. In other words, normalized data models can require more round trips to the server.

The application layer is responsible for managing the relationships in the normalized model.  The application will also need to make more requests against the database.

Now what would the normalized model look like in Couchbase?  Instead of calling it normalized, it is often called a referred model in Couchbase, and it would look like this:

{
    "_type": "person",
    "first_name": "Nic",
    "last_name": "Raboy",
    "address": "address1",
    "social_media": [
        {
            "type": "twitter",
            "url": "https://www.twitter.com/nraboy"
        },
        {
            "type": "mastodon",
            "url": "https://toot.cafe/@nraboy"
        }
    ]
}

Remember, the id is stored as meta information and most documents have a property to define what type of document it is.  Looking at the address property we are using a key to a different document.  The concept is the same as what MongoDB has with object ids, but in MongoDB the object id is a data type.

Looking at the document containing address data, we have something like this:

{
    "_type": "address",
    "city": "Mountain View",
    "state": "California"
}

The document key or id of the above document would be address1 to match what is expected in the other document.

Here is the kicker though.  The referred documents in Couchbase can be joined in a single server side operation through N1QL rather than forcing the application layer to take care of it.

Conclusion

What I could have probably summarized in a single sentence was that data modeling in MongoDB and data modeling in Couchbase is the same. It doesn’t matter if one uses BSON or JSON, the concepts apply through both. ¬†If you were to switch from MongoDB to Couchbase, everything you knew about MongoDB documents could be carried over.

The core differences come in regards to how you query the documents. It is much easier to query for data in Couchbase through N1QL and the other query strategies, regardless how you’ve chosen to model your documents.

If you’re interested in seeing a modeling and querying video I recorded, check it out here.

For more information on using Couchbase, check out the Couchbase Developer Portal which contains examples and other documentation.

The post Data Modeling NoSQL Documents in MongoDB vs Couchbase Server appeared first on The Couchbase Blog.

Categories: Architecture, Database

FakeIt Series 5 of 5: Rapid Mobile Development with Sync-Gateway

NorthScale Blog - Wed, 05/10/2017 - 13:21

Aaron Benton is an experienced architect who specializes in creative solutions to develop innovative mobile applications. He has over 10 years experience in full stack development, including ColdFusion, SQL, NoSQL, JavaScript, HTML, and CSS. Aaron is currently an Applications Architect for Shop.com in Greensboro, North Carolina and is a Couchbase Community Champion.

Aaron Benton

For our last post in the FakeIt¬†series we are going to explore how we can leverage FakeIt¬†+ Couchbase Server¬†+ Sync Gateway¬†to get our local environment up and running for mobile development. We will be doing this by using Docker¬†and docker-compose. Disclaimer: I am by no means a Docker expert, this is simply an example of what I’ve done to quickly setup a development environment and dataset.

Docker

We will be using two Docker Containers, one for Couchbase Server and one for Sync Gateway. We could define a docker-compose.yaml file that simply pulled from couchbase:latest and couchbase/sync-gateway:latest but there would still be a need for manual configuration and we want to be able to automate as much as possible for our application. To do that we will need to build our own containers from both of these adding our own scripts and configuration.

./docker-compose.yaml

version: '2'
services:
  fakeit-couchbase:
    build:
      context: ./.docker/couchbase/
    container_name: fakeit-couchbase
    ports:
      - "8091-8094:8091-8094"
      - "11210:11210"
    volumes:
      - ./:/app
  fakeit-syncgatway:
    build:
      context: ./.docker/sync-gateway/
    container_name: fakeit-syncgateway
    depends_on:
      - fakeit-couchbase
    ports:
      - "4984-4985:4984-4985"
    volumes:
      - ./:/app

Our docker-compose.yaml file is first building a container in the Dockerfile from ./.docker/couchbase/Dockerfile which looks like this.

./.docker/couchbase/Dockerfile

# start with couchbase
FROM couchbase:latest

# copy the configure script
COPY scripts/configure-node.sh /opt/couchbase

# execute the configure-node.sh script 
CMD ["/opt/couchbase/configure-node.sh"]

This Dockerfile is really only doing two things, copying a configuration script and executing that script. The configure-node.sh script looks like this.

./.docker/couchbase/scripts/configure-node.sh

set -m

/entrypoint.sh couchbase-server &

echo 'Waiting 20 seconds for Couchbase service to start'
sleep 20

# configure the cluster
echo 'Configuring Cluster'
/opt/couchbase/bin/couchbase-cli cluster-init -c localhost:8091 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=500 --service=data

# create the ecommerce bucket
echo 'Creating ecommerce bucket'
/opt/couchbase/bin/couchbase-cli bucket-create -c localhost:8091 -u Administrator -p password --bucket=ecommerce --bucket-eviction-policy=fullEviction --bucket-type=membase --bucket-priority=high --enable-index-replica=0 --bucket-port=11211 --enable-flush=1 --bucket-replica=1 --bucket-ramsize=200

echo 'Couchbase server is ready'

fg 1

 

The configure-node.sh script is doing a couple of things:

  1. Waiting for the Couchbase service to start so it can be configured
  2. Initializing the Cluster configuration
  3. Creating our ecommerce bucket

The Couchbase container is now built, the next container that needs to be built is the Sync Gateway container. By default the Sync Gateway container uses the walrus memory only sync gateway bucket. We will need to update this configuration by providing our own sync-gateway.json file so we can update the storage and access settings. Lastly, the Sync Gateway container only exposes port 4984, which is the public port, as this is for development purposes we will go ahead and expose port 4985 which is the administrator port.

./.docker/sync-gateway/Dockerfile

# Start with the base sync gateway
FROM couchbase/sync-gateway:latest

# Copy the sync-gateway.json into the container
COPY sync-gateway.json /opt/sync_gateway/sync-gateway.json

# Create Volume for data to persist
RUN mkdir -p /opt/sync_gateway/data

# Copy Entry Point
COPY scripts/entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]

# Copy the configure script
COPY scripts/configure-node.sh /opt/sync_gateway

# Configure the Sync Gateway and Start it
CMD ["/opt/sync_gateway/configure-node.sh"]

#  port 4984: public port
#  port 4985: admin port
EXPOSE 4984 4985

./.docker/sync-gateway/sync-gateway.json

{
 "interface": "0.0.0.0:4984",
 "adminInterface": "0.0.0.0:4985",
 "log": ["CRUD+", "REST+", "Changes+", "Attach+"],
 "CORS": {
   "Origin":[
     "http://localhost:8000",
    "*"
    ],
   "LoginOrigin":[
     "http://localhost:8000",
     "*"
   ],
   "Headers":["Content-Type"],
   "MaxAge": 1728000
 },
 "databases": {
   "ecommerce": {
     "server": "http://fakeit-couchbase:8091",
     "bucket": "ecommerce",
     "users": {
       "GUEST": {
         "disabled": false
       }
     },
     "sync": "function(doc, oldDoc) { channel(doc.channels); }"
   }
 }
}

./.docker/sync-gateway/scripts/configure-node.sh

#!/bin/bash

set -m

echo 'Waiting 20 seconds for Couchbase service to start and warm up'

sleep 20

echo 'Starting the Sync Gateway Service'
/entrypoint.sh sync_gateway /opt/sync_gateway/sync-gateway.json

echo 'Couchbase Sync Gateway is ready'

 

Again, this is for development purposes only. You should never allow access from anywhere to your adminInterface or enable GUEST access to your sync-gateway unless there is a very good reason to do so.

The structure of our app now looks like this:

app structure

Now that we have configured out Docker containers and setup our docker-compose.yaml file we need to build and start the containers. We do this by executing the following command from within our applications directory:

docker-compose up -d

Note: for the purposes of the screencast, the -d parameter to run the containers in detached mode is omitted.

blog5 example1

Models

Our containers are now started, the next thing we need to do before we generate our dataset is update the models to support the channels attribute.

users.yaml

This model will only be synced to user specific channels.

name: Users
type: object
key: _id
data:
  min: 1000
  max: 2000
  inputs: ./countries.json
properties:
  _id:
    type: string
    description: The document id built by the prefix "user_" and the users id
    data:
      post_build: `user_${this.user_id}`
  channels:
    type: array
    data:
      post_build: |
       return [ `channel-user-${this.user_id}` ];
  doc_type:
    type: string
    description: The document type
    data:
      value: "user"
...

products.yaml

Just for fun we will publish this model to a global channel all of our users will be subscribed to.

name: Products
type: object
key: _id
data:
  min: 500
  max: 1000
  inputs:
    - ./categories.csv
  pre_build: globals.current_category = faker.random.arrayElement(inputs.categories);
properties:
  _id:
    type: string
    description: The document id
    data:
      post_build: `product_${this.product_id}`
  channels:
    type: array
    data:
      build: |
       return [ `channel-products` ];
  doc_type:
    type: string
    description: The document type
    data:
      value: product
...

orders.yaml

This model will only be synced to user specific channels.

name: Orders
type: object
key: _id
data:
  dependencies:
    - products.yaml
    - users.yaml
  min: 5000
  max: 6000
properties:
  _id:
    type: string
    description: The document id
    data:
      post_build: `order_${this.order_id}`
  channels:
    type: array
    data:
      build: |
       return [ `channel-user-${this.user_id}` ];
  doc_type:
    type: string
    description: The document type
    data:
      value: "order"
...

Now that our models have been updated to support channels, we can generate our random dataset and push it into Couchbase Server through the Sync Gateway REST API. We tell FakeIt to do this by using the following command:

fakeit sync-gateway --server http://localhost:4984 --bucket ecommerce --verbose models/*

For development purposes we have allowed guest access to our Sync Gateway. However, if you have guest access disabled, you can still use FakeIt by specifying a username and password to an existing user by using the following command:

fakeit sync-gateway --server http://localhost:4984 --bucket ecommerce --username YOURUSERNAME --password YOURPASSWORD --verbose models/*

Before the generated dataset is output, FakeIt will authenticate against the Sync Gateway to retrieve the necessary session information.

blog5 example2

Testing

The next thing we need to do is to create a sync gateway user so that we can sync their documents locally. For this test, we are going to grab a random user document from our Users model and create a user from that document. In this example it will be user_1001, and we will create the user using the curl command:

curl --silent --show-error \
 -H "Content-Type: application/json; charset=UTF-8" \
 -H "Content-type: application/json" \
 -X PUT \
 -d '{"name":"Domenic81","password":"pgid_Tubn0qoEtZ","admin_channels":["channel-user-1001", "channel-products"]}' \
 'http://localhost:4985/ecommerce/_user/Domenic81'

 

blog5 example3

Application

We’ve created a small VueJS¬†project that utilizes PouchDB¬†to connect to the Sync Gateway and pull an authenticated users documents. Our application is going to just show the different types of documents available, their IDs and content.

// register highlight.js vue component
Vue.use(VueHighlightJS)

// create local ecommerce database
var db = new PouchDB('local_ecommerce', {
 auto_compaction: true,
 revs_limit: 3
});
// create remote ecommerce database
var remote_db = new PouchDB('http://localhost:4984/ecommerce/', {
 auth: {
   username: 'Domenic81',
   password: 'pgid_Tubn0qoEtZ'
 },
 skip_setup: true
});
// merge remote database with local database
PouchDB.sync(db, remote_db, { live: true, retry: true })

var app = new Vue({
 el: '#app',
 data: {
   current_type: '',
   items: [],
   id: '',
   current: JSON.stringify({}),
   types: [],
 },
 watch: {
   id(id) {
     if (id) {
       db.get(id).then((current) => this.current = JSON.stringify(current, null, 2))
     }
   },
   current_type(type) {
     this.id = ''
     // update the current item to be blank
     this.current = JSON.stringify({})

     // when `current_type` changes update the items list with the current types list
     this.updateItems(type)
       .then(() => {
         if (!this.id) {
           this.id = this.items[0]
         }
       })
   }
 },
 methods: {
   updateItems(type) {
     return db.query('ecommerce/by_doc_type', { key: type, reduce: false })
       .then((result) => {
         this.items = result.rows.map((item) => item.id)
       })
   },
   removeStyle(el) {
     setTimeout(() => {
       el.style = ''
     }, 1000)
   }
 },
 created() {
   // create the view to be queried
   var ddoc = {
     _id: '_design/ecommerce',
     views: {
       by_doc_type: {
         map: function(doc) {
           if (doc.doc_type) {
             emit(doc.doc_type);
           }
         }.toString(),
         reduce: '_count'
       }
     }
   }

   db.put(ddoc)
     // design document created
     // call the design document view index immediately to trigger a build
     .then(() => db.query('ecommerce/by_doc_type', { limit: 0 }))
     .catch((err) => { // it is fine if this fails and returns a document conflict that just means it doesnt need to be created
       if (err.status !== 409) {
         throw err;
       }
     })

     // get all of the doc types available
     .then(() => db.query('ecommerce/by_doc_type', { reduce: true, group: true }))
     .then((types) => {
       this.types = types.rows.map((type) => type.key)
       this.current_type = this.types[0]
     })
 }
})

 

blog5 example4

The entire sample application can be viewed at https://github.com/bentonam/fakeit-couchbase-mobile-example

Conclusion

Throughout this series you’ve seen how FakeIt¬†can take simple YAML models and generate vast amounts of fake data and send that data to multiple different destinations. Please checkout the repository, pull requests are welcome, we are always looking for improvements and enhancements to make the tool more useful for the community. I would also like to take this time to thank those who’ve made contributions to the project. The 1.0 version would not have been released without the help of ¬†Tyler Benton¬†(@tjbenton21), also Trevor Brindle¬†(@VinceKerrazzi), Jessica Kennedy¬†(@mistersender), Adam Burdette¬†(@RavenStorms619) and Brant Burnett¬†(@btburnett3)

Previous

Couchbase Champion

The post FakeIt Series 5 of 5: Rapid Mobile Development with Sync-Gateway appeared first on The Couchbase Blog.

Categories: Architecture, Database

Containerize a Node.js Application that Communicates with Couchbase Server

NorthScale Blog - Tue, 05/09/2017 - 16:00

Last month I had written a tutorial on the subject of containerizing a web application that talks to Couchbase Server.  This tutorial called, Deploy a Node.js with Couchbase Web Application as Docker Containers, helped you containerize a Node.js application, but this time around I figured it would be helpful to have a step by step video to avoid having any loose ends in the process.

The video below is a complete walkthrough on the process and can be applied on any Mac, Linux, or Windows machine that is running Docker.

If you need help with containerizing a Couchbase Server NoSQL cluster, check out a previous tutorial I wrote titled, Using Couchbase with Docker and Deploying a Containerized NoSQL Cluster.

To find more information on Couchbase, Docker, and any of the development SDKs, check out the Couchbase Developer Portal.

The post Containerize a Node.js Application that Communicates with Couchbase Server appeared first on The Couchbase Blog.

Categories: Architecture, Database

Deploying a Containerized Couchbase NoSQL Database Cluster with Docker ‚Äď Video Tutorial

NorthScale Blog - Thu, 05/04/2017 - 16:00

Docker is all the rage lately so I’ve decided to hop on this train and whip together some content regarding the containerization of Couchbase. ¬†I’ve seen a few tutorials that demonstrate the use of Couchbase with Docker, but I’ve decided to take a new spin at it by creating a video tutorial that walks through the steps of deploying a Couchbase NoSQL cluster.

If you’re interested in deploying a containerized Node.js web application that uses Couchbase containers, check out this article that I had previously written called,¬†Deploy a Node.js with Couchbase Web Application as Docker Containers.

To get more help with Docker, Couchbase Server, and any of the development SDKs, check out the Couchbase Developer Portal.

The post Deploying a Containerized Couchbase NoSQL Database Cluster with Docker – Video Tutorial appeared first on The Couchbase Blog.

Categories: Architecture, Database

Get to know our Couchbase Community Meet Zijun Yang

NorthScale Blog - Thu, 05/04/2017 - 13:38

Meet this week’s Couchbase Community Champion, Zijun Yang

Zijun Yang

Tell us about yourself and what you do in your daily role?
I am a System Architect in a startup company named Paclane in Peking, China. I have 10 years experience in system integration, and 8 years experience in the Identity Access Management area. I am now responsible for designing system architecture and researching new technologies that we might use, such as react, react native, Couchbase, IoT,ETL tools, web security, and so on. I like to read Couchbase forums and blogs to find out how users use Couchbase, as well as  contribute to the community.. I think it is the best way to learn new technologies.

Where are you based and what do you like to do in your spare time?
I am currently based in Sinkiang, China, and work remotely from home. In my spare time I like to play with my two little daughters at home, and other times I like to research internet products and their related business models.

What made you choose Couchbase?
We needed to change schema frequently, and we wanted to extend the database easily. Couchbase offered big data support, mobile offline support, XDCR for data sync between the data centers, IoT support and full-text search in the future, and easy data query with N1QL.

What one feature of Couchbase do you use and what should people know about it?
N1QL is powerful, flexible, and easy. We use N1QL-prepared REST API to package our business logic for data query, and we use Sync Gateway to control data write.

What have you accomplished using Couchbase?
We started to use Couchbase Mobile by PhoneGap, query data by Couchbase view for web application, and we use Elasticsearch plugin to transfer data to Elasticsearch, and then query data by Elasticsearch for some complex query.¬†When N1QL was announced, we removed Elasticsearch,¬†and we used¬†N1QL to rewrite all view query and Elasticsearch query¬†(we don’t use full-text search yet). It works great.

If you could have three wishes for Couchbase what would they be?
My three wishes would be:
1. More events in China. (I participated in a Couchbase event in China in 2015, and then no other events were found in China. fFew people know Couchbase here).
2. More introduction about how people use N1QL and the Couchbase data platform.
3. Increase more integration in ETL and big data ecology.

What is your favorite book or author, and why?
My favourite book in school was ‚ÄúThe Fifth Discipline‚Ä̬†by Peter M. Senge because it showed¬†me a brand-new view about organization and the way to think about ¬†problems.¬†My favourite book today¬†is ‚ÄúSapiens: A Brief History of Humankind‚Ä̬†by Yuval Noah Harari. It also shows me a brand-new view to better understand the world.

 

Couchbase Champion

The post Get to know our Couchbase Community Meet Zijun Yang appeared first on The Couchbase Blog.

Categories: Architecture, Database

Data Sharing with Couchbase Lite between iOS Apps and App Extensions

NorthScale Blog - Wed, 05/03/2017 - 17:30

This post discusses how to use Couchbase Lite as an embedded database to share data between your iOS App and iOS App Extension.  App Extensions implement a specific task or functionality that can be exposed to other apps on the device or to the Operating System. In this post, we will walk through an example of how a Today Extension can be used with a Couchbase Lite embedded database in standalone mode.

NOTE:  We will be discussing Couchbase Mobile v1.4 which is the current production release. There is a newer Developer Preview version 2.0 of Couchbase Mobile

Background

Apple supports many kinds of App Extensions¬†, each of which enables functionality that is relevant to a particular subsystem on the device.¬† In this post, we will discuss how Couchbase Lite can be used with a Today Extension. This type of extension, also known as a “Widget”, appears in the Today view of the Notification Center and allows users get quick updates. You can learn more about how App Extensions work in¬†the Apple Developer documents.

I’ll assume you’re familiar with developing iOS Apps in Swift and have a basic understanding of integrating Couchbase Lite into your iOS App. This Getting Started guide is a great place to begin.  If would like to read up more on Couchbase, refer to the resources at the end of this post.

Couchbase Lite

Couchbase Lite is an embedded database that runs on devices. It can be used in several deployment modes. It can be used as a standalone embedded database or it can be used in conjunction with a remote Sync Gateway that would allow it to sync data across devices. In this post, we will use Couchbase Lite in standalone deployment mode. In this post, we will not go over the details of integrating with Couchbase Lite. The Getting Started With Couchbase Lite blog is a good place to get started on that.

Demo App

Download the Demo Xcode project from the Github repo . We will use this app as an example in the rest of the blog.

git clone git@github.com:couchbaselabs/couchbase-lite-ios-standalone-sampleapp.git

This is a simple Task List app that allows users to add, edit, delete tasks. A user can mark tasks as completed. A Today Extension is bundled with the app that shows the top 2 tasks right in your notification center without the need to open the app. The user can mark tasks as completed right from the notification center.

All tasks are stored in a local Couchbase Lite database. This implies that the Container app and the extension will both need access to the database.

App Architecture

App Extensions are not standalone apps. They are bundled within an App, referred to as a ‚ÄúContainer App‚ÄĚ.¬† Although App Extensions are bundled in the Container app, they run independent of the Container App in a separate process . App Extensions¬†are launched by other apps that need the extension‚Äôs functionality. The App that launches the App Extension is¬†referred to as¬†the “Host App“.¬† The extension‚Äôs UI is displayed in the context of the Host App.

Although the Container App and the corresponding extension are independent processes running in their own sandbox, they can share data through a Shared Container.

The Shared Container can be set up by registering a unique App Group and enabling it for for both the container app and the extension. You will learn about setting this up in the next section.

App Extension Architecture

Configuring App Groups

Refer to the Apple Developer Docs to learn more about configuring App Groups in your iOS app

Open the CBLiteAppExtension.xcworkspace of the demo app project that you downloaded.

  • Open the ‚ÄúCapabilities‚ÄĚ tab of CBLiteApp Container App target . You should see the App Group group.com.example.CBLiteSharedData enabled.
Enabling App Group

Enabling App Group for App

 

 

 

 

 

 

 

 

 

  • Open the ‚ÄúCapabilities‚ÄĚ tab of CBLiteTaskExtension target. You should see the same App Group, group.com.example.CBLiteSharedData enabled.

Configuring App group in Extension

 

 

 

 

 

 

 

 

 

Enabling the App Group capabilities for a target adds it to the Entitlements files for the container app and it’s¬†extension.

  • In addition , you will have to add the App Groups feature to your App Id registered in the Apple Developer Portal.
Configuring App Group for App in Apple Developer Portal

Configuring App Group for App in Apple Developer Portal

 

App Walkthrough

Build and Run the App by choosing the App Target ‚ÄúCBLiteAppExtension‚ÄĚ. Now switch to the Today view of the Notification Center

  • Add your new Extension Widget to the Today View as shown below

 

Add Today Widget

  • Switch to the App and add couple of tasks tapping on the ‚Äú+‚ÄĚ button.¬†Switch to the corresponding Today widget. You will notice that the tasks that you added are displayed in the widget.

Add Task

  • Mark a task as ‚ÄúCompleted‚ÄĚ by tapping on the task.¬†Now switch back to the Container app. You will notice the completion status of the tasks is updated to correspond to actions taken from the Today widget.

Today Extension Update Task

 

  • Similarly edits or delete a¬†task in¬†the Container app by swiping the task entry. Switch to the Today widget. You will see the updated task list.

Force Touch

  • If your device supports 3D Touch, you can do a force touch on the app icon on home screen to reveal the Top Tasks extension right there from the icon and you can interact with it. Pretty cool !

Force Touch

  • Finally, terminate the Container App. Switch to the Today Widget and update the Completed status of a task. Relaunch the app. The tasks will be listed with the updated status.

Container App Termination

 

Code Walkthrough

Tasks are stored in a local Couchbase Lite embedded database. This implies that both the container app and the Today Extension need access to the database. As discussed earlier, the way to share data between the container app and the extension is through the Shared Container. This implies that the Couchbase Lite database must be located in the Shared Container.

The code to enable this is straightforward.

Open the DatabaseManager.swift file.

The DatabaseManager is a singleton class that handles basic database management functions.

Locate appGroupContainerURL function

This function constructs the path to the shared container folder.

// 1. Get URL to shared group container
        let fileManager = FileManager.default
        guard let groupURL = fileManager
            .containerURL(forSecurityApplicationGroupIdentifier: "group.com.example.CBLiteSharedData") else {
                return nil
        }
        
        let storagePathUrl = groupURL.appendingPathComponent("CBLite")
        let storagePath = storagePathUrl.path
        
        // 2: Create a folder in the shared container location with name"CBLite"
        if !fileManager.fileExists(atPath: storagePath) {
            do {
                try fileManager.createDirectory(atPath: storagePath,
                                                withIntermediateDirectories: false,
                                                attributes: nil)
            } catch let error {
                print("error creating filepath: \(error)")
                return nil
            }
        }
        
        return storagePathUrl

  1. Return the URL to the shared group container. Group containers are stored in ~/Library/Group Containers/<application-group-id>
  2. Create a folder named CBLite in the shared group container.

Locate configureCBManagerForSharedData function

do {
            // 1. Set the file protection mode for the Couchbase Lite database folder
            let options = CBLManagerOptions(readOnly: false, fileProtection: Data.WritingOptions.completeFileProtectionUnlessOpen)
            let cblpoptions = UnsafeMutablePointer<CBLManagerOptions>.allocate(capacity: 1)
            cblpoptions.initialize(to: options)
            
            if let url = self.appGroupContainerURL() {
                // 2. Initialize the CBLManager with the directory of the shared container
                _cbManager = try CBLManager.init(directory: url.relativePath, options: cblpoptions)
            }
            
            return true
        }
        catch {
             return false
            
        }

  1. Create CBLManagerOptions options object with the appropriate file protections. A value of completeFileProtectionUnlessOpen implies that read/write access to the file is restricted unless file is open
  2. Initialize the CBLManager with the path to the shared container. When the database is created, it will be created in the shared container.That’s it! The Couchbase Lite database is now in the Shared Container that both the App and the Extension can read and write to.
What Next?

Explore the rest of the demo sample code to understand how documents are added, edited and deleted. Specifically look at the TaskPresenter.swift file. This is where all the interactions with the Couchbase Lite database is handled.

If you have any questions, feel free to reach out to me on Twitter. If you would like to suggest improvements, submit a pull request to the GitHub repo. You can learn more about integrating with Couchbase Lite in this Getting started with Couchbase Lite blog. The Couchbase Forums are another great place to post your questions.

While this post discusses the use of Couchbase Lite in standalone mode, stay tuned for an upcoming post that will enhance the capability to include synchronization with the cloud.

The post Data Sharing with Couchbase Lite between iOS Apps and App Extensions appeared first on The Couchbase Blog.

Categories: Architecture, Database

Sample Application with Spring and Angular: GitTalent

NorthScale Blog - Wed, 05/03/2017 - 15:30

One of the first big projects I was involved in after I started at Couchbase last year was helping to create a sample application that would be demonstrated at Couchbase Connect 2016.

Sample application

It was an enormous team effort to create something that would be simple enough to show in a keynote demo, while being complex enough to show off the powerful new features of Couchbase Server.

I think the demo went really well, but you don’t have to take my word for it. You can watch the demo in action below (the demo starts about 26 minutes in).

By the way, all the sessions from Couchbase Connect 2016 were recorded are are available for you to watch.

Our goal was to create a tool that we could use to empathize with users (aka dog fooding), engage potential customers with a comprehensive sample application to help get them comfortable with Couchbase, and, ultimately, to show off Couchbase at our signature conference.

GitTalent sample application resources

The lead on the project (Laurent Doguin) has since left Couchbase (just after I finally learned to spell his name correctly!), but one of the last things he did was make a publicly available copy of the GitTalent sample application and write an incredible, detailed book about its design and code.

I use the word “book” because what he wrote is exactly that. It’s far too long to do it justice in a single blog post. So instead, here’s a link to the GitTalent book. It’s in AsciiDoc format, but GitHub does a pretty good job of rendering it.

Note that this demo was carried out completely live on stage. The specific repo used in the demo is private (for security reasons), but here’s a screenshot of the commit from the demo:

Live git commit to sample application during presentation

The entire project is available on GitHub in the GitTalent repository. My contributions were mostly in writing Angular, but I also wrote some Java (believe it or not!). So if you have any questions about the project, I’ll do my best to help. Please leave a comment below, or ask on the Couchbase Forums.

The post Sample Application with Spring and Angular: GitTalent appeared first on The Couchbase Blog.

Categories: Architecture, Database

Deploy a Golang Web Application and Couchbase as Docker Containers

NorthScale Blog - Tue, 05/02/2017 - 16:01

In my development series regarding Docker containers for the web application developer, I had already gone over deploying Java and Node.js applications as containers alongside Couchbase Server Containers.  This time around I thought it would be cool to deploy a Golang web application as a Docker container alongside Couchbase.

The process is very similar to what I had already demonstrated in the previous tutorials, but it is worth exploring. ¬†We’re going to explore containerizing an application that I had explained in a previous tutorial on the topic of URL shortening services.

If you’re interested, you can check out¬†Use Docker to Deploy a Containerized Java with Couchbase Web Application and¬†Deploy a Node.js with Couchbase Web Application as Docker Containers, depending on your language preferences.

The Requirements

There is only one dependency that must be met to be successful with this guide. ¬†You need Docker installed on a host machine. ¬†You don’t need Golang installed or Couchbase Server since that is the beauty of the Docker Engine.

While you don’t have to, I would recommend becoming familiar with the application in question. ¬†To learn about the code behind the application, check out the tutorial I wrote called,¬†Create a URL Shortener with Golang and Couchbase NoSQL.

Establishing a Docker Project for the Golang Application

The source code to the application, and what we’ll be bundling, can be seen below. ¬†You should save it as a Go source code file such as¬†main.go.

package main

import (
	"encoding/json"
	"log"
	"net/http"
	"os"
	"time"

	"github.com/couchbase/gocb"
	"github.com/gorilla/mux"
	"github.com/speps/go-hashids"
)

type MyUrl struct {
	ID       string `json:"id,omitempty"`
	LongUrl  string `json:"longUrl,omitempty"`
	ShortUrl string `json:"shortUrl,omitempty"`
}

var bucket *gocb.Bucket
var bucketName string

func ExpandEndpoint(w http.ResponseWriter, req *http.Request) {
	var n1qlParams []interface{}
	query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE shortUrl = $1")
	params := req.URL.Query()
	n1qlParams = append(n1qlParams, params.Get("shortUrl"))
	rows, _ := bucket.ExecuteN1qlQuery(query, n1qlParams)
	var row MyUrl
	rows.One(&row)
	json.NewEncoder(w).Encode(row)
}

func CreateEndpoint(w http.ResponseWriter, req *http.Request) {
	var url MyUrl
	_ = json.NewDecoder(req.Body).Decode(&url)
	var n1qlParams []interface{}
	n1qlParams = append(n1qlParams, url.LongUrl)
	query := gocb.NewN1qlQuery("SELECT `" + bucketName + "`.* FROM `" + bucketName + "` WHERE longUrl = $1")
	rows, err := bucket.ExecuteN1qlQuery(query, n1qlParams)
	if err != nil {
		w.WriteHeader(401)
		w.Write([]byte(err.Error()))
		return
	}
	var row MyUrl
	rows.One(&row)
	if row == (MyUrl{}) {
		hd := hashids.NewData()
		h := hashids.NewWithData(hd)
		now := time.Now()
		url.ID, _ = h.Encode([]int{int(now.Unix())})
		url.ShortUrl = "http://localhost:12345/" + url.ID
		bucket.Insert(url.ID, url, 0)
	} else {
		url = row
	}
	json.NewEncoder(w).Encode(url)
}

func RootEndpoint(w http.ResponseWriter, req *http.Request) {
	params := mux.Vars(req)
	var url MyUrl
	bucket.Get(params["id"], &url)
	http.Redirect(w, req, url.LongUrl, 301)
}

func main() {
	router := mux.NewRouter()
	cluster, _ := gocb.Connect("couchbase://" + os.Getenv("COUCHBASE_HOST"))
	bucketName = os.Getenv("COUCHBASE_BUCKET")
	bucket, _ = cluster.OpenBucket(bucketName, "")
	router.HandleFunc("/{id}", RootEndpoint).Methods("GET")
	router.HandleFunc("/expand/", ExpandEndpoint).Methods("GET")
	router.HandleFunc("/create", CreateEndpoint).Methods("PUT")
	log.Fatal(http.ListenAndServe(":12345", router))
}

The only difference between the code above and the previous tutorial on the subject is the use of the os.Getenv("COUCHBASE_HOST") and os.Getenv("COUCHBASE_BUCKET") commands.  This will allow us to define the Couchbase connection information at container runtime via environment variables rather than hardcoding it into the application.

If you had Couchbase Server running, you could launch the web application by doing:

env COUCHBASE_HOST=localhost COUCHBASE_BUCKET=default go run *.go

Of course the project would need to be in your $GOPATH, but it would be accessible from http://localhost:12345 as defined in the code.

The goal here is not to run this project locally, but via a Docker container.  Create a Dockerfile file next to the main.go file or whatever you called it.  The Dockerfile file should contain the following:

FROM golang:alpine

RUN apk update && apk add git

COPY . /go/src/app/

WORKDIR /go/src/app

RUN go get -d -v
RUN go install -v

CMD ["app"]

As a base we’ll be using the official Golang Alpine Linux image for Docker, but we’ll be customizing it. ¬†During build time we are installing Git, copying the¬†main.go file to the image, installing the project dependencies found in the¬†import section, and installing the application. ¬†At runtime we are executing the installed application.

To build an image from our custom Docker blueprint, we would execute the following:

docker build -t golang-project .

After all the compile-time steps complete we’ll be left with a¬†golang-project Docker image. ¬†Before we try to run this image, we need to worry about Couchbase. ¬†While we could run Couchbase outside a container, where is the fun in that?

Creating a Custom Couchbase Server Docker Image

Just like with Golang, an official Docker image exists for Couchbase Server. ¬†While it is a perfectly acceptable solution, it is not an automated solution. ¬†This means that you’ll have to manually configure the database after the container starts. ¬†We probably want to avoid that.

Instead, create a directory somewhere on your computer with a Dockerfile file and configure.sh file in it.  The plan is to create a Docker blueprint that will execute the configuration script for us.

Open the Dockerfile file and include the following:

FROM couchbase

COPY configure.sh /opt/couchbase

CMD ["/opt/couchbase/configure.sh"]

We’re basically just copying the script into the base image and running it. ¬†The real magic happens inside of the script.

Open the configure.sh file and include the following:

set -m

/entrypoint.sh couchbase-server &

sleep 15

curl -v -X POST http://127.0.0.1:8091/pools/default -d memoryQuota=512 -d indexMemoryQuota=512

curl -v http://127.0.0.1:8091/node/controller/setupServices -d services=kv%2cn1ql%2Cindex

curl -v http://127.0.0.1:8091/settings/web -d port=8091 -d username=$COUCHBASE_ADMINISTRATOR_USERNAME -d password=$COUCHBASE_ADMINISTRATOR_PASSWORD

curl -i -u $COUCHBASE_ADMINISTRATOR_USERNAME:$COUCHBASE_ADMINISTRATOR_PASSWORD -X POST http://127.0.0.1:8091/settings/indexes -d 'storageMode=memory_optimized'

curl -v -u $COUCHBASE_ADMINISTRATOR_USERNAME:$COUCHBASE_ADMINISTRATOR_PASSWORD -X POST http://127.0.0.1:8091/pools/default/buckets -d name=$COUCHBASE_BUCKET -d bucketType=couchbase -d ramQuotaMB=128 -d authType=sasl -d saslPassword=$COUCHBASE_BUCKET_PASSWORD

sleep 15

curl -v http://127.0.0.1:8093/query/service -d "statement=CREATE PRIMARY INDEX ON `$COUCHBASE_BUCKET`"

fg 1

Couchbase Server has a RESTful interface which will allow us to complete the configuration wizard through a script.  The above commands will define instance specs and services, administrative credentials, and a Bucket.

You’ll notice things like $COUCHBASE_ADMINISTRATOR_USERNAME within the script. ¬†Just like in the Golang application we are leveraging environment variables to prevent having to hardcode values into our image. ¬†These values can be defined during the container deployment.

More information on provisioning a Couchbase Server container can be found in a previous article that I wrote on the subject.

Finally we can build this custom Couchbase Server image.  From the Docker Shell, execute the following:

docker build -t couchbase-custom .

The above command will leave us with a couchbase-custom image.

Deploying the Containerized Microservice

There are a few ways to deploy the images that we built. ¬†We’ll explore two of them here.

Given what we know about our images, we can deploy our containers with the following commands:

docker run -d \
    -p 8091-8093:8091-8093 \
    -e COUCHBASE_ADMINISTRATOR_USERNAME=Administrator \
    -e COUCHBASE_ADMINISTRATOR_PASSWORD=password \
    -e COUCHBASE_BUCKET=default \
    -e COUCHBASE_BUCKET_PASSWORD= \
    --network="docker_default" \
    --name couchbase \
    couchbase-custom

The above command will deploy Couchbase Server while mapping the necessary ports to the host operating system. ¬†We’re passing each environment variable in with the command and defining an operating network. ¬†The network is important because you want both your database and application to be on the same container network.

docker run -d \
    -p 12345:12345 \
    -e COUCHBASE_HOST=couchbase \
    -e COUCHBASE_BUCKET=default \
    --network="docker_default" \
    --name golang 
    golang-project

The above command will deploy our Golang image while mapping the correct ports again. ¬†For the host you’ll notice that we’re using¬†couchbase which is actually the name of our other container. ¬†The container names also act as host names.

The commands you just saw are a very manual way to do things in my opinion.  Instead you can create a Docker Compose file to handle this for you.

Create a docker-compose.yml file somewhere on your computer with the following:

version: '2'
 
services:
    couchbase:
        image: couchbase-custom
        ports:
            - 8091:8091
            - 8092:8092
            - 8093:8093
        environment:
            - COUCHBASE_ADMINISTRATOR_USERNAME=Administrator
            - COUCHBASE_ADMINISTRATOR_PASSWORD=password
            - COUCHBASE_BUCKET=default
            - COUCHBASE_BUCKET_PASSWORD=
 
    golang:
        image: golang-project
        ports:
            - 12345:12345
        environment:
            - COUCHBASE_HOST=couchbase
            - COUCHBASE_BUCKET=default
        restart: always

Everything in a Compose file will be on the same network.  To deploy each service, execute the following:

docker-compose run -d --service-ports couchbase
docker-compose run -d --service-ports golang

If you’re familiar with Docker Compose you’ll be familiar with¬†docker-compose up, but we can’t use that here. ¬†Couchbase Server doesn’t have a mechanism to tell us it is ready, so we don’t want the Golang application to try to connect before it is configured. ¬†Definitely not a big deal when it comes to container deployment.

Conclusion

You just saw how to create a Golang microservice as a Docker container that communicates to Couchbase Server, also running within a Docker container. ¬†By turning your application into a Docker image it becomes very easy to distribute because you won’t have to worry about the environment you’re trying to deploy to. ¬†As long as the Docker Engine is available you can deploy your application. ¬†A useful example of this would be in continuous deployment of containers using Jenkins.

Want to see how to containerize your Java application?  Check out this tutorial I wrote.  I also have a tutorial for containerizing a Node.js application here.

Want more information on using Golang with Couchbase? Check out the Couchbase Developer Portal for documentation and examples.

The post Deploy a Golang Web Application and Couchbase as Docker Containers appeared first on The Couchbase Blog.

Categories: Architecture, Database

Improved SDK Authentication Methods ‚Äď Couchbase 5.0

NorthScale Blog - Mon, 05/01/2017 - 22:15

Couchbase Server 5.0 delivers some great new authentication features that require some minor changes to your client connection code.  The specific changes to the SDK are outlined in this post and also refer to changes made in the pre-release April Developer Build as well as upcoming Beta releases.

See¬†Matthew’s Introduction to Role-Based Access¬†Control: Part 1¬†&¬†Part 2

In a nutshell, bucket-based passwords (and passwordless buckets) are now a thing of the past. ¬†Instead, we’ve provided fine-grained¬†role level access control to buckets, data and all services. ¬†You are now able to create a user with specific roles and access rights,¬†then¬†connect as that user (with password) in your client.

There are three specific patterns to using these new features, I review each option in this post.

As this is based on pre-production server code, examples may change.  Please also note that you need to have the latest SDK libraries for your language, supported at the time of writing are Java, .NET, Node.js, PHP, Python, Go and C.

User Named Buckets

If you continue to use the basic URL connection string for a bucket without a username, there are some assumptions that the client now makes.

First, it will assume you have a user with the same name as your bucket.  Migrating users from earlier versions will appreciate this as all they have to do is provide a password with the connection string as shown in this Python example:

>>> db = Bucket("couchbase://localhost/tyler",password="tyler123")

(This simplified usage is limited to use for just for Python, PHP and C libraries)

The above assumes not only that there is a bucket named tyler but also a user named tyler who has a bucket access role enabled.

Passwordless buckets are no longer allowed, but with one exception: migrated buckets.  It is impossible to create a new user without a password but the migration process will create users, without passwords, for buckets that formerly did not have one.

As above, the user will have the same name as the bucket and no password.  In that case you can pass an empty password string with the connection string.

After the migration, new buckets and users will all require passwords, so this is a good time to change your approach going forward.

Passing Username and Password With Bucket Name

The new authentication paradigm going forward has two ways of managing authenticated connections.  The simplest approach is to adjust the connection parameters to include both username and password, this is easily done in Python, :

>>> db = Bucket("couchbase://localhost/travel-sample", username="tyler", password="tyler123")

However, this is a very simplistic approach.  There is a more powerful option that uses the new Authenticator class to manage authentication against, potentially, multiple buckets.

Authenticator Class  for Handling Users and Buckets

The “proper” way to handle authentication is to de-couple it from the Bucket URI altogether and pass it to the Cluster connection when needed.

Here the Authenticator initialization takes a given Cluster and authenticates against it.  In these examples, you see that the pattern across languages is the same.  The Authenticator can then be passed to other Clusters as desired.

Python

from couchbase.cluster import Cluster
from couchbase.cluster import PasswordAuthenticator

cluster = Cluster('couchbase://localhost')
  
authenticator = PasswordAuthenticator('username', 'password')
cluster.authenticate(authenticator)
bucket = cluster.open_bucket('bucket-name')

Java

Cluster cluster = CouchbaseCluster.create();
cluster.authenticate("username", "password");
Bucket bucket = cluster.openBucket("bucket-name");

.NET

var cluster = new Cluster(new ClientConfiguration 
  { Servers = new List<Uri> { new Uri("http://localhost") }});
var authenticator = new PasswordAuthenticator("username", "password");
cluster.Authenticate(authenticator);
var bucket = cluster.OpenBucket("bucket-name");

PHP

$authenticator = new \Couchbase\PasswordAuthenticator();
$authenticator->username('username')->password('password');

$cluster = new \Couchbase\Cluster('couchbase://localhost');
$cluster->authenticate($authenticator);

Node.js

var couchbase = require('couchbase');
var cluster = new couchbase.Cluster('couchbase://localhost/');
cluster.authenticate('username', 'password');
var bucket = cluster.openBucket('bucket-name');

 

Next Steps

Notice that the main design change in your applications is to not use a bucket specific password.  Instead, you need to have a user with an adequate role that has access to particular buckets.

With only a couple lines of code change you easily take advantage of the role-based access control features of Couchbase Server 5.0.

Upgrade to the latest developer build and upgrade your client libraries to start trying it out.

Join us in the forums with any questions!

 

The post Improved SDK Authentication Methods – Couchbase 5.0 appeared first on The Couchbase Blog.

Categories: Architecture, Database

Triggering Data Synchronization from the Cloud using Urban Airship Push Notifications

NorthScale Blog - Mon, 05/01/2017 - 07:03
Introduction

In this post I’ll show a sample Android application built with Couchbase Lite that uses an Urban Airship push notification to trigger synchronizing data from the cloud to a mobile device. Here “the cloud” will be a Sync Gateway instance setup in a test configuration. (Couchbase Lite and Sync Gateway are both part of the Couchbase Mobile stack. You can read about Couchbase Mobile here.)

Background

Couchbase Mobile synchronizes data using unidirectional replications. You can set replications to run and end immediately once the local data is up-to-date (“one-shot”), or to listen indefinitely for new changes (“continuous”).

There are a few reasons you might want to use a service like Urban Airship to trigger synchronization instead of setting up a continous replication. For example, a continuous “pull” replication (one that retrieves data from the cloud) needs to keep a network connection open. An open connection will necessarily cause some amount of drain on the device battery, even while inactive. Ideally, an application would only open a connection when there exists changes to transfer.

Beyond that, push notification services like Urban Airship typically have a number of advanced features for addressing groups of devices. UA can give you great flexibility in managing when operations happen. Furthermore, if your app already uses a push service, that already requires its own open network connection. It makes sense to use that to trigger Couchbase Lite instead of doubling up the networking.

This example will show how to integrate Urban Airship. I’ll walk through all the necessary pieces, so you can build a fully working application.

What you’ll need
  • An Urban Airship account
  • A Firebase account (requires a Google Login account)
  • Android Studio or the equivalent

Note: You can read about setting up Urban Airship in this blog post.

The Application

Here’s a look at the application in action. The upper part of the image shows the app running. The lower part shows a push notification sent from the command line. When the app receives the push notification, it puts up a progress spinner, fires off a one-shot pull replication, then displays the document changes received in a text view.

To understand this example, we’ll take a look at four classes, some of the “glue” that brings them together, and show how to test the results.

Classes Required by Urban Airship

To use Urban Airship (UA), we need to implement two classes, a subclass of Autopilot, and a subclass of AirshipReceiver.

Autopilot

UA has to be initialized before use. You can do that in two ways, either call takeOff in the application‚Äôs onCreate method or by creating a subclass of Autopilot. I chose to do the latter. As we’ll see, UA has an interesting way of automatically using our subclass.

The class is simple. I don’t want to display anything in the notification bar. I use this class to turn off user notifications, and that’s all.

Here’s the full listing of CBAutoPilot.java.

package com.couchbase.cblite.android.cbpushsync;

import com.urbanairship.Autopilot;
import com.urbanairship.UAirship;

public class CBAutoPilot extends Autopilot {
    @Override
    public void onAirshipReady(UAirship airship) {
        airship.getPushManager().setUserNotificationsEnabled(false);
    }
}

To use this class, UA employs a technique unique to Android. You indicate which class to use with an entry in your Android manifest file. For my case, the entry looks like this (placed inside the application section):

<meta-data android:name="com.urbanairship.autopilot"
            android:value="com.couchbase.cblite.android.cbpushsync.CBAutoPilot"/>

This is where a bit of Android magic happens. UA includes a Content Provider class to automate the initilization. Content Providers are guaranteed to get instantiated before the application’s onCreate method gets called. UA uses that to include code to read this meta-data entry, instantiate the subclass, and call the onAirshipReady method.

AirshipReceiver

With UA initialized and ready to go, we need to provide the hooks to capture and process notifications. To do that, implement a subclass of AirshipReceiver. Again, the class is straightforward.

Here’s the full listing of CBAirshipReceiver.java.

package com.couchbase.cblite.android.cbpushsync;

import android.content.Context;
import android.support.annotation.NonNull;
import android.util.Log;

import com.urbanairship.AirshipReceiver;
import com.urbanairship.push.PushMessage;

public class CBAirshipReceiver extends AirshipReceiver {
    private static final String TAG = "CBAirshipReceiver";

    @Override
    protected void onPushReceived(Context context, PushMessage message, boolean notificationPosted) {
        Log.i(TAG, "Received push message. Alert: " + message.getAlert() + ". posted notification: " + notificationPosted);

        switch(message.getAlert()) {
            case "start":
                CBLHelper.getInstance().startReplication();
                break;
            case "stop":
                CBLHelper.getInstance().stopReplication();
                break;
            case "reset":
                CBLHelper.getInstance().reset();
                break;
            default:
                break;
        }
    }

    @Override
    protected void onChannelCreated(@NonNull Context context, @NonNull String channelId) {
        Log.i(TAG, "Channel created. Channel Id:" + channelId + ".");
    }

    @Override
    protected void onChannelUpdated(@NonNull Context context, @NonNull String channelId) {
        Log.i(TAG, "Channel updated. Channel Id:" + channelId + ".");
    }

    @Override
    protected void onChannelRegistrationFailed(Context context) {
        Log.i(TAG, "Channel registration failed.");
    }
}

When we get to testing the app, we’ll see how UA lets you set the alert text. I used that here provide a few different remote operations. An alert can start and stop replications. I also included the ability, by sending “reset”, to delete the database. I did that because once you replicate the data, it’s persistent on the device. Shutting down the app doesn’t reset the local database. Rather than always having to add new documents through Sync Gateway to show replication working, I added the ability to wipe the device database and start over.

You’ll notice a few other overridden methods above. Those methods are all declared abstract in the AirshipReceiver class, so we have to include at least a minimal implementation. It turns out, though, that we need to know the device channel ID to send it a message. This channel ID changes often. In a future post, I’ll show how I register the ID so a server-side application can send to the device. For now, it’s useful to log the ID so we can extract it and use it in testing.

The Application Classes

The hooks above get UA up and running, trap notifications, and trigger actions based on their contents. Now let’s take a look at the application itself.

The app consists of one Activity set up so we can see the contents of documents as they get pulled. The other class wraps some of the Couchbase Lite functionality in a helper.

The Main Activity

In the main activity, I want to show a progress bar while replications happen, then display the set of documents sent.

Here’s the full listing of MainActivity.java

package com.couchbase.cblite.android.cbpushsync;

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
import android.support.v7.widget.Toolbar;
import android.util.Log;
import android.view.View;
import android.widget.ProgressBar;
import android.widget.TextView;

import com.couchbase.lite.Database;
import com.couchbase.lite.Document;
import com.couchbase.lite.DocumentChange;
import com.couchbase.lite.replicator.Replication;
import com.couchbase.lite.replicator.ReplicationState;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.IOException;

public class MainActivity extends AppCompatActivity {
    private static String TAG = MainActivity.class.getSimpleName();
    private static final ObjectMapper mapper = new ObjectMapper();

    private CBLHelper cblHelper = CBLHelper.getInstance();
    private TextView documentView;
    private ProgressBar progressBar;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
        setSupportActionBar(toolbar);

        documentView = (TextView) findViewById(R.id.documentView);
        progressBar = (ProgressBar) findViewById(R.id.progressBar);

        cblHelper.initialize(
            new Database.ChangeListener() {
                @Override
                public void changed(Database.ChangeEvent event) {
                    Log.i(TAG, "Database change called: count - " + event.getChanges().size());

                    if (!event.isExternal()) {
                        return;
                    }

                    String json = "";

                    for (final DocumentChange change : event.getChanges()) {
                        if (!change.isCurrentRevision()) {
                            continue;
                        }

                        Document changedDoc = cblHelper.getExistingDocument(change.getDocumentId());

                        if (changedDoc == null) continue;

                        try {
                            json += mapper.writeValueAsString(changedDoc.getProperties());
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    }

                    final String text = json;

                    runOnUiThread(new Runnable() {
                        @Override
                        public void run() {
                            documentView.setText(text);
                        }
                    });
                }
            },
            new Replication.ChangeListener() {
                @Override
                public void changed(Replication.ChangeEvent event) {
                    Log.i(TAG, "Replication change called: " + event.toString());

                    if (event.getError() != null) return;

                    if (event.getTransition() == null) return;

                    ReplicationState dest = event.getTransition().getDestination();

                    final int replicationProgress =
                            ((dest == ReplicationState.STOPPING ||
                              dest == ReplicationState.STOPPED) ?
                                    View.INVISIBLE : View.VISIBLE);

                    runOnUiThread(new Runnable() {
                        @Override
                        public void run() {
                            progressBar.setVisibility(replicationProgress);
                        }
                    });
                }
            }
        );
    }
}

The core of the code here consists of the two change listeners, one for the database, and one for replications. Each listener interface defines a single method changed.

The Document changes listener lets us track the actual document updates. A replication may change more than one document at a time. The change event returns a list of document IDs. Looping over the document IDs, we retrieve each document directly, translate the JSON contents into a string, and append it to the text to display. The change listeners are called on a background thread. To finish out, we need to manipulate the UI elements on the main thread. This is easily done with the runOnUiThread method of the Activity.

The other listener allows us to display a busy-wait spinner (indefinite progress bar) depending on the Replication state. You can read more about monitoring replication state here.

The Database Helper Class

I often wrap database functions in a helper class. Since we only need one instance of the helper, I use a singleton pattern. Just like UA, I want the helper available before the main application starts. I use the same technique of putting the code in a Content Provider.

Here’s the full listing of CBLHelper.java.

package com.couchbase.cblite.android.cbpushsync;

import android.content.ContentProvider;
import android.content.ContentValues;
import android.database.Cursor;
import android.net.Uri;
import android.os.Handler;
import android.os.Looper;
import android.support.annotation.NonNull;
import android.support.annotation.Nullable;
import android.util.Log;

import com.couchbase.lite.CouchbaseLiteException;
import com.couchbase.lite.Database;
import com.couchbase.lite.Document;
import com.couchbase.lite.Manager;
import com.couchbase.lite.android.AndroidContext;
import com.couchbase.lite.replicator.Replication;

import java.io.IOException;
import java.net.URL;

public class CBLHelper extends ContentProvider {
    public static final String TAG = "Push Sync";

    private static String DB = "db";
    //private static String syncGateway = "http://localhost:4984";
    private static String syncGateway = "http://10.0.2.2:4984";
    private static URL syncGatewayURL;

    private Manager manager;
    private Database database;
    private Replication puller;

    private Database.ChangeListener databaseChangeListener;
    private Replication.ChangeListener replicationChangeListener;

    private static CBLHelper instance;

    public static CBLHelper getInstance() { return instance; }

    @Override
    public boolean onCreate() {
        instance = this;

        enableLogging();

        try {
            syncGatewayURL = new URL(syncGateway + "/" + DB);
            manager = new Manager(new AndroidContext(getContext()), Manager.DEFAULT_OPTIONS);
        } catch (IOException ex) {
            ex.printStackTrace();
        }

        return true;
    }

    public void closeDatabase() {
        database.close();
    }

    public void startReplication() {
        puller.start();
    }

    public void stopReplication() { puller.stop(); }

    public void reset() {
        stopReplication();

        try {
            database.delete();
        } catch (CouchbaseLiteException ex) {
            ex.printStackTrace();
        }

        initialize(databaseChangeListener, replicationChangeListener);
    }

    public void initialize(Database.ChangeListener dbl, Replication.ChangeListener rl) {
        try {
            database = manager.getDatabase(DB);
        } catch (CouchbaseLiteException ex) {
            ex.printStackTrace();
        }

        database.addChangeListener(dbl);
        databaseChangeListener = dbl;

        puller = database.createPullReplication(syncGatewayURL);
        puller.addChangeListener(rl);
        replicationChangeListener = rl;
    }

    public Document getExistingDocument(String documentID) {
        return database.getExistingDocument(documentID);
    }

    // Logging

    private void enableLogging() {
        Manager.enableLogging(TAG, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG_SYNC_ASYNC_TASK, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG_SYNC, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG_QUERY, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG_VIEW, Log.VERBOSE);
        Manager.enableLogging(com.couchbase.lite.util.Log.TAG_DATABASE, Log.VERBOSE);
    }

    // Helper function to dispatch on UI thread
    private void runOnUiThread(Runnable runnable) {
        new Handler(Looper.getMainLooper()).post(runnable);
    }

    // Required overrides

    @Nullable
    @Override
    public Cursor query(@NonNull Uri uri, @Nullable String[] projection, @Nullable String selection, @Nullable String[] selectionArgs, @Nullable String sortOrder) {
        return null;
    }

    @Nullable
    @Override
    public String getType(@NonNull Uri uri) {
        return null;
    }

    @Nullable
    @Override
    public Uri insert(@NonNull Uri uri, @Nullable ContentValues values) {
        return null;
    }

    @Override
    public int delete(@NonNull Uri uri, @Nullable String selection, @Nullable String[] selectionArgs) {
        return 0;
    }

    @Override
    public int update(@NonNull Uri uri, @Nullable ContentValues values, @Nullable String selection, @Nullable String[] selectionArgs) {
        return 0;
    }
}

You can see this class just forms a thin wrapper. We retain an instance of the class during onCreate which becomes our singleton. Some other basic setup happens, like enabling logging and creating a database manager. I wrote an initialize method used to actually open the database, prepare the pull replication, and attach the two change listeners. Recall this is called from our Activity class. (Note: You can have more than one change listener for each type. Couchbase keeps a list of all added listeners, not just the latest one.) The other database methods just provide simple shortcuts.

A content provider has several mandatory methods to implement. We need to provide stubs for those, which you see in the last five methods.

The Android Manifest

Finally, just to show how this ties together and the various parts get exposed, I’ve included the Android manifest file for the project.

Here’s the complete listing of AndroidManifest.xml.

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.couchbase.cblite.android.cbpushsync">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:supportsRtl="true"
        android:theme="@style/AppTheme">
        <meta-data android:name="com.urbanairship.autopilot"
            android:value="com.couchbase.cblite.android.cbpushsync.CBAutoPilot"/>
        <activity
            android:name=".MainActivity"
            android:label="@string/app_name"
            android:theme="@style/AppTheme.NoActionBar">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
        <receiver android:name=".CBAirshipReceiver"
            android:exported="false">

            <intent-filter>
                <action android:name="com.urbanairship.push.CHANNEL_UPDATED" />
                <action android:name="com.urbanairship.push.OPENED" />
                <action android:name="com.urbanairship.push.RECEIVED" />
                <action android:name="com.urbanairship.push.DISMISSED" />

                <category android:name="${applicationId}" />
            </intent-filter>
        </receiver>
        <provider
            android:authorities="${applicationId}.dbhelper"
            android:exported="false"
            android:enabled="true"
            android:name=".CBLHelper" />
    </application>

</manifest>

The UA library comes with its own manifest file. The Android build system integrates all the different manifest files to create the final one. The UA library manifest file is where you’ll find the entries for the Content Provider that bootstraps the library.

Sync Gateway

To try the app out, I used Sync Gateway with the built-in walrus database. Walrus is an in-memory database usually used just for testing. This saves the trouble of setting up a back-end Couchbase Server instance.

Here’s the complete listing of the Sync Gateway configuration file.

{
  "log": ["HTTP+"],
  "adminInterface": "127.0.0.1:4985",
  "interface": "127.0.0.1:4984",
  "CORS": {
    "origin":["*"],
    "loginorigin":["*"],
    "headers":["Content-Type"],
    "maxAge": 1728000
  },
  "databases": {
    "db": {
      "server": "walrus:",
      "users": { "GUEST": {"disabled": false, "admin_channels": ["*"] } }
    }
  }
}

This tells Sync Gateway to listen only on the localhost interface, accept any cross origin requests, and enables the special GUEST with access to all channels. It’s a pretty good general purpose configuration to start with to make sure everything’s working.

You might have noticed we use 10.0.2.2 as the IP address for Sync Gateway in the Android app. The standard emulator that comes with Android Studio maps this automatically to the hosting machine’s localhost. If you use a different emulator (Genymotion is another popular one), be sure to look up what IP address to use, since they differ.

If you want to learn more about setting up Sync Gateway on your development machine, or how to work with it from the command line, take a look at this blog series.

Testing

To test everything and record the animation shown at the beginning of the blog, I run both the Android emulator and Sync Gateway on one machine. I then post push notifications to UA through their REST API.

To get something interesting prepared, I first add a document to Sync Gateway. Here’s a shell command to do that.

$ curl -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "test" : "UA", "channels": "public" }' http://localhost:4984/db/doc

(Read more about this here).

With a new document in the database on Sync Gateway, and the app up and running, all that’s left is to trigger a pull replication to see the action.

Here’s the curl command to send a “start” signal. Recall I pull the value from the “alert” part of the notification to trigger actions.

$ curl https://go.urbanairship.com/api/push -u 'appKey:appMasterSecret' -X POST -H 'Content-Type: application/json' -H 'Accept: application/vnd.urbanairship+json; version=3;' -d '{ "audience": { "android_channel": "3110bc81-1c7b-4ef8-a7a3-7c56aab19f08" }, "notification" : { "alert": "start" }, "device_types": "all" }'

You’ll need to provide your own application key and application master secret above. You also need to look at the log files to see the channel value to enter for “android_channel”. That changes fairly regularly. Unfortunately UA doesn’t do much to indicate this. If you find push notifications don’t seem to be having any effect, check the channel value.

After posting to UA, you should see a response like this.

{"ok":true,"operation_id":"2a6c0edf-e5d0-4b4d-b762-ad5c4478c291","push_ids":["1951019a-2b7f-4f1b-8888-e87047a1feec"],"message_ids":[],"content_urls":[]}

And, finally, to reset to a blank database to try again, you can post this.

$ curl https://go.urbanairship.com/api/push -u 'appKey:appMasterSecret' -X POST -H 'Content-Type: application/json' -H 'Accept: application/vnd.urbanairship+json; version=3;' -d '{ "audience": { "android_channel": "3110bc81-1c7b-4ef8-a7a3-7c56aab19f08" }, "notification" : { "alert": "reset" }, "device_types": "all" }'

Conclusion

That’s it for getting our Android app prepared to perform replications based on push notifications. Be sure and check out this post on monitoring the Sync Gateway changes feed. I’ll be writing another post to show a more sophisticated version, combining monitoring the changes feed with triggering client actions soon.

Postscript

Download Couchbase and Sync Gateway here. See our documentation for how to add Couchbase Lite to a project.

Check out more resources on our developer portal and follow us on Twitter @CouchbaseDev.

You can post questions on our forums. And we actively participate on Stack Overflow.

Hit me up on Twitter with any questions, comments, topics you’d like to see, etc. @HodGreeley.

The post Triggering Data Synchronization from the Cloud using Urban Airship Push Notifications appeared first on The Couchbase Blog.

Categories: Architecture, Database

New to Couchbase Server 5.0: GSI replicas and rebalancing

NorthScale Blog - Sun, 04/30/2017 - 10:29

Couchbase Server 4.0 introduced Global Secondary Indexes (GSI) to support the N1QL query engine.  Now, with Couchbase Server 5.0 (as of the April developer build), we have added the ability to automatically replicate and rebalance these indexes.

At first, GSIs were managed very manually:

  • To ensure High Availability, the end-user manually configured multiple indexes with the same definition and specified which nodes of the cluster they should be deployed on. ¬†If one node failed, the end-user had¬†to manually re-create the index that was lost.
  • When adding and removing nodes either for scaling out, upgrading or after a failure, the end-user had¬†to manually move and recreate the indexes on the nodes that were coming and going.

I’m extremely proud to report that both of these key operational tasks are now automatically built-in to Couchbase Server and I will walk you through them here.

GSI Replication

One of the most frequent requests that we heard from customers after the introduction of GSIs in 4.0 was for automatic management of replication: “I just want to tell you how many replicas and have you figure it out”

Now you can:

CREATE INDEX `idx_airportname` ON `travel-sample`(`airportname`) WITH {"num_replica":1};

With one simple command, Couchbase will:

  • Create two identical indexes
  • Deploy and build those indexes on separate nodes
  • Honor the Rack-Zone Awareness configuration

If not enough nodes are available, you are quickly presented with an error:

[
  {
    "code": 5000,
    "msg": "GSI CreateIndex() - cause: Fails to create index.  There are not enough indexer nodes to create index with replica count of 2",
    "query_from_user": "CREATE INDEX `idx_airportname` ON `travel-sample`(`airportname`) WITH {\"num_replica\":2};"
  }
]

Pretty cool eh?

When using “num_replica”, you are relying on Couchbase to determine the proper index layout within the cluster. ¬†For even more fine-grained control, specify the exact nodes you wish a single index to be replicated across:

CREATE INDEX `idx_airportname` ON `travel-sample`(`airportname`) WITH {"nodes":["ec2-35-167-251-49.us-west-2.compute.amazonaws.com:8091", "ec2-54-69-121-55.us-west-2.compute.amazonaws.com:8091"]};

Voila!

These indexes will now be automatically load balanced by the query engine and any failure (so long as at least one matching index remains) will not cause any disruption to the queries.

GSI Rebalancing

It’s not enough just to have automatic replica¬†creation, we also needed to add automatic management in the form of rebalancing.

Again, as of the April developer build of Couchbase Server 5.0, the index service is now “rebalance aware”:

  • When¬†an index node is removed from the cluster, its indexes will be redistributed to the remaining nodes automatically. ¬†However, adding a new index node to the cluster will not cause the indexes to be re-distributed. ¬†This is by design to prevent excess movement of indexes and also to preserve the placement that an administrator may have specified. ¬†This redistribution is something that might be added in a later release if needed.
  • If an index node fails and is replaced, the appropriate replicas will be re-created. ¬†Similarly, those index replicas are¬†re-created on the remaining nodes of a cluster if a rebalance is performed without replacing the failed node.

I encourage you to give this all a try and find that it “just works”! ¬†Click here to download Couchbase Server.

The post New to Couchbase Server 5.0: GSI replicas and rebalancing appeared first on The Couchbase Blog.

Categories: Architecture, Database

Save Captured Images in a NativeScript Angular Application to Couchbase

NorthScale Blog - Thu, 04/27/2017 - 15:45

I get a particular set of questions quite a bit when it comes to mobile applications, the first being how to save captured images, and the second being how to sync them between devices or a remote database. Since I’m a huge fan of NativeScript for Android and iOS development, I thought it would be great to share one of many possible examples toward solving this problem.

We’re going to see how to capture images in a NativeScript application using Angular and save those captured images to Couchbase Lite, an embedded device level database.

Going forward, it is important to note that there are many ways to save images or any other file data within a mobile application. For example you could save them in a database or you can save them on the file system. We’re going to be saving them directly into the database in this example, but do your research on what strategy will work best for you.

NativeScript Couchbase Photos

In the above animated image we are presented with the ability to capture images.  Things are a little different in a simulator, but on a device the camera will open.  After capturing an image it will be presented on the screen and saved into Couchbase Lite as a base64 formatted string.

The Requirements

There aren’t many requirements when it comes to being successful with this guide. ¬†At a minimum you’ll need the following:

  • NativeScript CLI
  • Android SDK for Android and Xcode for iOS

To be able to build for Android you’ll need the Android SDK and to be able to build for iOS you’ll need Xcode, which is only available on macOS computers. For this particular guide, no remote instance of Couchbase Server is necessary.

Creating a Fresh NativeScript with Angular Project

With the prerequisites installed and ready to go, we need to create a fresh project somewhere on our computer.

From the NativeScript CLI, execute the following:

tns create photo-project --ng

The above command will create a new Angular project, hence the --ng flag.

Before we start developing, we will need to install the Couchbase plugin for NativeScript.  This can be done by executing the following:

tns plugin add nativescript-couchbase

While we can technically start developing the application, there is one more order of business to take care of. On iOS it is a requirement to explain each of the permissions being used. Being able to take photos or use the gallery is a permission request, so we have to explain.

Open the project’s¬†app/App_Resources/iOS/Info.plist file and include the following:

<key>NSPhotoLibraryUsageDescription</key>
<string>Photo Library Access Warning</string>

The above will prevent any errors during compile time or run time.  Now we can safely start developing our Android and iOS mobile application.

Developing the Core Application Logic and User Interface

Before we start adding our own code, let’s strip out a bunch of the boilerplate template code that ships when creating a new NativeScript with Angular project.

This is going to be a single page application so we need to remove any of the pre-existing navigation routes. ¬†Start by opening the project’s¬†app/app.routing.ts file and make it look like the following:

import { NgModule } from "@angular/core";
import { NativeScriptRouterModule } from "nativescript-angular/router";
import { Routes } from "@angular/router";

const routes: Routes = [];

@NgModule({
    imports: [NativeScriptRouterModule.forRoot(routes)],
    exports: [NativeScriptRouterModule]
})
export class AppRoutingModule { }

The only thing we did was remove the routes from the¬†routes array. ¬†Technically we can remove this file, but it is easier just to strip out what we’re not going to use.

Now open the project’s¬†app/app.module.ts file and make the TypeScript look like the following:

import { NgModule, NO_ERRORS_SCHEMA } from "@angular/core";
import { NativeScriptModule } from "nativescript-angular/nativescript.module";
import { AppRoutingModule } from "./app.routing";
import { AppComponent } from "./app.component";


@NgModule({
    bootstrap: [
        AppComponent
    ],
    imports: [
        NativeScriptModule,
        AppRoutingModule
    ],
    declarations: [
        AppComponent
    ],
    providers: [],
    schemas: [
        NO_ERRORS_SCHEMA
    ]
})
export class AppModule { }

Notice in the above we’ve removed any reference to the MVC files that were previously routes. ¬†This includes HTML, TypeScript, and the Angular service that was provided.

When it comes to adding new content, we’re going to focus on the TypeScript logic first, then move into the HTML UI. Open the project’s¬†app/app.component.ts file and include the following:

import { Component, OnInit } from "@angular/core";
import { Couchbase } from "nativescript-couchbase";
import * as Camera from "camera";
import * as ImageSource from "image-source";

@Component({
    selector: "ns-app",
    templateUrl: "app.component.html",
})
export class AppComponent implements OnInit {

    public database: any;
    public images: Array<any>;

    public constructor() {
        this.database = new Couchbase("image-database");
        this.database.createView("images", "1", function(document, emitter) {
            if(document.type && document.type == "image") {
                emitter.emit(document._id, document);
            }
        });
        this.images = [];
    }

    public ngOnInit() {
        let rows = this.database.executeQuery("images");
        for(let i = 0; i < rows.length; i++) {
            this.images.push(ImageSource.fromBase64(rows[i].image));
        }
    }

    public capture() {
        Camera.takePicture({ width: 300, height: 300, keepAspectRatio: true, saveToGallery: false }).then(picture => {
            let base64 = picture.toBase64String("png", 70);
            this.database.createDocument({
                "type": "image",
                "image": base64,
                "timestamp": (new Date()).getTime()
            });
            this.images.push(picture);
        }, error => {
            console.dump(error);
        });
    }

}

A lot is happening in the above code so we’re going to break it down.

We had previously downloaded the Couchbase plugin for NativeScript, but now we need to import it along with other things:

import { Couchbase } from "nativescript-couchbase";
import * as Camera from "camera";
import * as ImageSource from "image-source";

The camera functionality is already included with NativeScript, we just need to import it. ¬†When it comes to manipulating the camera data, the¬†ImageSource is what we’ll use.

Within the AppComponent class we have two variables:

private database: any;
public images: Array<any>;

The database will hold our open Couchbase Lite instance and the images will hold all saved images that will be presented on the screen.

Within the constructor method we open our database, create a view that we can later query, and initialize the array that will store our images.

public constructor() {
    this.database = new Couchbase("image-database");
    this.database.createView("images", "1", function(document, emitter) {
        if(document.type && document.type == "image") {
            emitter.emit(document._id, document);
        }
    });
    this.images = [];
}

The view that we’re creating will return all NoSQL documents that have a property called¬†type that is set to¬†image which represents to us that it is one of the possible images for displaying on the screen.

Because data should never be loaded in the constructor method, we take things to the ngOnInit method:

public ngOnInit() {
    let rows = this.database.executeQuery("images");
    for(let i = 0; i < rows.length; i++) {
        this.images.push(ImageSource.fromBase64(rows[i].image));
    }
}

The ngOnInit method triggers after the constructor method and it will query the view that had been previously created. Each document saved will have a property called image that contains base64 image data. This model is based on our design.

After obtaining the base64 data it is converted into an ImageSource and added to our array to be displayed on the screen.

public capture() {
    Camera.takePicture({ width: 300, height: 300, keepAspectRatio: true, saveToGallery: false }).then(picture => {
        let base64 = picture.toBase64String("png", 70);
        this.database.createDocument({
            "type": "image",
            "image": base64,
            "timestamp": (new Date()).getTime()
        });
        this.images.push(picture);
    }, error => {
        console.dump(error);
    });
}

The above capture method is called via a button press in our HTML. It will launch the camera with a few settings defined.

Upon a successful capture, the picture will be converted to base64 and a NoSQL document will be created along with various information sitting next to the base64 data.

Not so bad right?

Now we want to take a look at the HTML markup that goes with this logic. ¬†Open the project’s¬†app/app.component.html file and include the following:

<ActionBar title="{N} Couchbase Photos">
    <ActionItem text="Capture" ios.position="right" (tap)="capture()"></ActionItem>
</ActionBar>
<StackLayout>
    <Image *ngFor="let image of images" [src]="image"></Image>
</StackLayout>

In the above HTML we have an actionbar with a button that will trigger the camera.  Within the core content of the page we have a loop that will go through each image in the array and display it on the screen.

Conclusion

You just saw how to create a basic image capturing application with NativeScript and Angular that saves image data directly into Couchbase Lite NoSQL documents as base64 encoded strings. Next time around we’re going to see how to synchronize this image data between devices and a remote database instance.

In the meantime, check out this other tutorial on using NativeScript with Couchbase titled, Key-Value Operations in Couchbase Mobile via NativeScript and Angular.

Want more help with Couchbase for Android and iOS? Check out the Couchbase Developer Portal for documentation and examples.

The post Save Captured Images in a NativeScript Angular Application to Couchbase appeared first on The Couchbase Blog.

Categories: Architecture, Database

A Comparison of SQL and NoSQL to Simplify Your Database Decision

NorthScale Blog - Thu, 04/27/2017 - 13:45

Tony Branson is a self-proclaimed tech geek, with a passion for ScaleArc’s disruptive technology innovation in database load balancing. He has a passion for dissecting tech topics such as transparent failover, centralized control, ACID compliance, database scalability, and downtime effects. On his days off, he can be found watching sci-fi movies, rock climbing or volunteering.

Have you been investing your time, effort, and resources in building ETL procedures that keep migrating data from one database to another? Is your RDBMS fully equipped to deal with unstructured and non-traditional data? With Big Data becoming the hottest segment of database technology, what’s your game plan to stay on top of the ever-evolving technologies?

nosql vs sql overview 1

However you implement ETL processes, you must decide how to structure your data and what technologies to use. To help you make an informed decision that is right for you, let’s start from the basics.

What is SQL?

SQL is a language that facilitates communication with relational database management systems, most of which have their exclusive proprietary extensions. SQL can do everything from accessing and manipulating databases to inserting records and creating views.

What is NoSQL?

NoSQL encompasses a wide range of database technologies that are designed to cater to the demands of modern apps. NoSQL systems make it easy to deploy and store a wide range of data types, and they excel in performance ‚Äď until you need data consistency and start applying techniques found in DBMSs that slow performance.

The Key Differences Between NoSQL and SQL

NoSQL SQL Storage NoSQL encompasses a host of database types ranging from graph and key-value to document and columnar, and each has a different data storage mechanism. Data is typically stored in a relational model where columns contain data points and rows comprise of all the information concerning a single entity. Flexibility Since schemas are dynamic in nature, information can be updated on the fly. In SQL, every record conforms to a predefined schema where the columns must be determined and locked before the data can be entered and it cannot be amended later without going offline and modifying the entire database. ACID Compliance NoSQL emphasizes performance over data integrity and most NoSQL systems compromise on ACID compliance for performance, so organizations use NoSQL for data types not impacted by consistency. SQL databases default to enabling ACID compliance though most offer options to favor performance over data integrity for some operations (e.g., asynchronous replication between sites can risk data loss during failure). Access Access is allowed in well-defined and narrow patterns which make performance and scalability dependable and expected. Not known beforehand and hence requires assumptions that are then translated into index definitions.  

 

 

Addressing the Big Question ‚Äď To SQL or NoSQL?

Your decision should be based on your immediate and long-term business requirements in terms of data needs, performance, and data types because there is no one-size-fits-all solution when it comes to database technology. While NoSQL comes with the advantages of speed and scalability, SQL remains the only choice for applications that require ACID compliancy. If your business is not going to experience significant growth in the near future and your data is structured, SQL is the right choice for your business. But if you desire rapid processing of real-time data and you don’t have transactional data to protect, NoSQL is your go-to solution.

Most organizations will need a combination of both ‚Äď both have their use cases in today‚Äôs digital business demands. Enterprises that use both concurrently are sure to deliver better business outcomes and maximize their ROI but they must apply them appropriately and they must deliver 100% uptime despite sudden traffic spikes. These situations show the value of database load balancing software that lets you handle high loads at peak performance levels without needing any modifications at the app level.

Database load balancing is valuable for addressing some of the scaling and performance challenges that can come with SQL databases. The software’s capabilities include:

  • Response time-based routing to ensure peak performance at all times
  • Surge queue feature to avoid overloading of the database server
  • Query routing to prevent unexpected outages
  • Connection pooling and multiplexing to facilitate fast and easy access, year round
  • Read/write split support to facilitate optimum utilization of available servers
  • App-transparent failover to improve application availability during database failures

With database load balancing software, enterprises get the best advantages of SQL and NoSQL databases, so they never have to rethink their database decision.

This post is part of the Couchbase Community Writing Program

The post A Comparison of SQL and NoSQL to Simplify Your Database Decision appeared first on The Couchbase Blog.

Categories: Architecture, Database

Create a Continuous Deployment Pipeline with Node.js and Jenkins

NorthScale Blog - Tue, 04/25/2017 - 16:05

Previously I had written about using Jenkins for continuous deployment of Java applications, inspired by a keynote demonstration that I had developed for Couchbase Connect 2016. ¬†I understand that Java isn’t the only popular development technology that exists right now. ¬†Node.js is a very popular technology and a perfect candidate to be plugged into a continuous deployment pipeline using Jenkins.

We’re going to see how to continuously deploy a Node.js application with Jenkins based on changes made to a GitHub repository.

So let’s figure out the plan here. ¬†We’re going to be using an already existing Node.js repository that I had uploaded to GitHub a while back. ¬†When changes are made to this repository, Jenkins will build the application and deploy or run the application. ¬†Because of the nature of Node.js, the build process will consist of making sure the NPM modules are present.

The Requirements

There are a few software requirements that must be met in order to be successful with this guide.  They are as follows:

Since this is a Node.js pipeline, of course we’ll need it installed. ¬†However, since Jenkins is a Java application, we’ll also need Java installed. ¬†My sample application does use Couchbase, but that won’t be the focus of this guide. ¬†However, if you’re using the same application I am, you’ll need Couchbase Server installed.

All software listed should reside on the same host.  In a production environment you will probably want them dispersed across multiple machines.

Installing and Configuring Couchbase Server as the NoSQL Database

At this point you should have already downloaded Couchbase Server. ¬†After installing it and configuring it, you’ll need to create a Bucket called¬†restful-sample and that Bucket should have a primary index.

For instructions on configuring Couchbase and getting this Bucket created, check out a previous tutorial I wrote on the subject.  It is actually the tutorial that went with creating this Couchbase, Express Framework, Angular, and Node.js (CEAN) stack application.

With Couchbase ready to go, we can focus on configuring Jenkins and creating our workflow.

Configuring Jenkins with the Necessary Plugins

You should have already downloaded Jenkins by now. ¬†If you haven’t go ahead and obtain the WAR file from the Jenkins website.

To start Jenkins, execute the following command from your Command Prompt or Terminal:

java -jar jenkins.war -httpPort=8080

This will make Jenkins accessible from a web browser at http://localhost:8080. ¬†Upon first launch, you’ll be placed in a configuration wizard.

Jenkins Configuration Part 1

The first screen in this configuration wizard will ask you for the password that Jenkins generates.  Find it in the location presented on the screen.

The second screen will ask you which plugins you’d like to install.

Jenkins Configuration Part 2

For now, we’re going to install the suggested plugins. ¬†We’ll be installing extra plugins later.

The third screen will ask us to create our first administrative user. ¬†Technically, the generated password you’re using is an administrative user, but you may want to create a new one.

Jenkins Configuration Part 3

After you create a user, Jenkins is ready to go.  However, we are going to need another plugin and it can vary depending on how we wish to build and deploy the Node.js application.

From the main Jenkins screen, choose to Manage Jenkins to see a list of administration options.

Manage Jenkins

What we care about is managing the available plugins.  After choosing Manage Plugins we want to search for and install a plugin by the name of Post-Build Script.

Install Jenkins Post-Build Script Plugin

This plugin allows us to execute shell commands or scripts after the build stage has completed. ¬†In this example we’ll be building and deploying to the same host, we can run everything locally via shell commands. ¬†In a production environment you might want to use the SSH plugin to migrate the code to a remote server and run it there.

With the plugins available, let’s create our continuous deployment workflow for Node.js in Jenkins.

Creating a Jenkins Continuous Deployment Workflow for Node.js

Just to reiterate, our goal here is to create a workflow that will pull a project from GitHub, build it by installing all the dependencies, and deploy it by running it on a server, in this case our local machine.

Start by creating a new item, otherwise known as a new job or workflow.

Jenkins Node.js Freestyle Project

We’re going to be creating a¬†Freestyle Project, but you can give it any name you want. ¬†There are three things that need to be done on the next screen.

The source of our workspace will come from GitHub.  In your own project it can come from elsewhere, but for this one we need to define our source control information.

Jenkins Node.js Source Control

The GitHub project is one that I had previously created and written about, like mentioned before.  The project can be found at:

https://github.com/couchbaselabs/restful-angularjs-nodejs

Now in a production environment you’ll probably want to set up GitHub hooks to trigger the job process, but since this is all on localhost, GitHub won’t allow it. ¬†Instead we’ll be triggering the job manually.

Jenkins Node.js Build Step

After configuring the source control section we’ll need to configure the build step. ¬†For Node.js, building only consists of installing dependencies, but you could easily have unit testing and other testing in this step as well. ¬†In my previous Java example, the build step had a little more to it. ¬†In this Node.js example we have the following:

npm install

Finally we get to define what happens after the project is built.

Jenkins Node.js Post Build Step

In this example we will be deploying the application locally on our machine.  Probably not the case in your production scenario.

So you’ll notice in our post-build step we have the following commands:

npm stop
npm start

Before starting the application we are stopping any already running instance of it.  Once stopped we can start the new version.  However, where do these stop and start tasks come from?

"scripts": {
    "start": "forever start app.js",
    "stop": "forever stopall"
}

The above was taken from the GitHub project’s¬†package.json file. ¬†Each task starts and stops a¬†forever script for Node.js.

Go ahead and try to run the job choosing¬†Build Now from the list of options. ¬†It should obtain the project, install the dependencies, and make the project available at http://localhost:3000. ¬†Just make sure Couchbase Server is running for this project, otherwise you’ll get errors.

Conclusion

You just saw how to use Jenkins to continuously deploy your Node.js applications based on changes that have been made in GitHub. ¬†A similar version of this guide was created for Java applications called,¬†Create a Continuous Deployment Pipeline with Jenkins and Java, which is worth reviewing if you’re a Java developer.

If you’re interested in using Jenkins to deploy your Node.js application as Docker containers, check out a previous tutorial that I wrote on the subject.

Want more information on using Node.js with Couchbase NoSQL?  Check out the Couchbase Developer Portal for documentation and examples.

The post Create a Continuous Deployment Pipeline with Node.js and Jenkins appeared first on The Couchbase Blog.

Categories: Architecture, Database