Skip to content

Software Development News: .NET, Java, PHP, Ruby, Agile, Databases, SOA, JavaScript, Open Source

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Data structures with Couchbase and .NET (video)

NorthScale Blog - 5 hours 46 min ago

In February, I wrote a blog about using data structures with .NET (and .NET Core): List, Queue, and Dictionary.

Now, I’ve created a video to show the same concepts in action.

How to use Couchbase Data Structures with .NET

The source code used in this video is available on GitHub. Note that the source code uses .NET Core, but this should work just the same in .NET.

Thanks for watching!

If you have any questions about Couchbase and .NET, please ask away in the Couchbase .NET Forums. Also check out the Couchbase Developer Portal for more information on the .NET SDK and Couchbase in general.

You can also contact me at Twitter @mgroves.

The post Data structures with Couchbase and .NET (video) appeared first on The Couchbase Blog.

Categories: Architecture, Database

Performance Enhancements to N1QL in the Couchbase 5.0 March Developer Build

NorthScale Blog - 7 hours 13 min ago

N1QL in Couchbase has come a long way since it was first introduced in Couchbase Server 4.0.  In Couchbase 5.0, things are taken to the next level in terms of performance.  In terms of the March 2017 Developer build of Couchbase 5.0, there are performance enhancements to N1QL in the flavor of index projection, enhancements to COUNT and DISTINCT, and the much requested ORDER BY, LIMIT, and OFFSET operators.

So what in specific was done to enhance all of these areas and how can we make use of the changes?

Let’s take index projection for example. ¬†When creating an index, you can create one with any number of properties. ¬†For example, take the following index:

CREATE INDEX idx ON default(type, firstname, lastname);

The above statement will create a covering index on the default Bucket for the type, firstname, and lastname properties of any given document.

Now let’s say we created the following N1QL query to retrieve a few documents with the¬†idx index we had created:

SELECT firstname
FROM default
WHERE type = 'person'

The above query would use the¬†idx index and return only the¬†firstname¬†property for every document that matches. ¬†The concept of querying this way is nothing new, however, what happens¬†behind the scenes has changed. ¬†You’ll notice that even though¬†our index has many keys, we’re only interested in a subset, or in this case two¬†keys.

So what is happening and why is this important?

In previous versions of Couchbase all keys of the index were taken into consideration regardless if only a subset were used.  As a result, more network, CPU, and memory were needed to accommodate what was happening.  Now this is not the case.

So how do you know index projection is happening?

Do an¬†EXPLAIN on the query that you’re running:

EXPLAIN SELECT firstname
FROM default
WHERE type = 'person'

In the results you should see something regarding index_projection that looks like the following:

...
"index_projection": {
    "entry_keys": [
        0,
        1
    ]
},
...

The entry_keys property will change based on your query.  For example, what if we add one WHERE condition like so?:

SELECT firstname
FROM default
WHERE type = 'person' AND lastname = 'Nic'

In the above scenario, we would get an EXPLAIN result that looks like the following:

...
"index_projection": {
    "entry_keys": [
        0,
        1,
        2
    ]
},
...

Now the above query wasn’t an index projection because we used all keys in our covering index.

Creating proper indexes paired with index projection can really help in overall performance and scaling your Couchbase Server cluster.

Index projection wasn’t the only performance enhancement made in the March 2017 build right? ¬†That is correct, there is more!

Let’s take the¬†COUNT(DISTINCT) operation for example. ¬†Now let’s use that operation in the following query:

EXPLAIN SELECT COUNT(DISTINCT type)
FROM default;

In the results you’ll notice that it is using¬†IndexCountDistinctScan2 and what it is doing is storing all¬†type in the index and processing the distinct values. ¬†While it happens in the indexer in Couchbase 5.0, it previously happened in the N1QL service in prior editions. ¬†By offloading¬†this operation in the indexer, we can experience significant performance gains.

Similarly, take the OFFSET, LIMIT, and ORDER BY operators that can be used in N1QL queries.  Take the following query for example:

EXPLAIN SELECT firstname
FROM default
WHERE type = 'person'
LIMIT 1
OFFSET 1
ORDER BY firstname;

You’ll notice that the¬†LIMIT, ORDER BY,¬†and¬†OFFSET operators will appear in the indexer. ¬†Prior to 5.0, the LIMIT operator appeared in the indexer, but now the others do as well. ¬†This is a huge win because in previous versions of Couchbase if you were to offset the results, N1QL would get all X number of results, and drop everything before the offset.

For more help with N1QL, check out the Couchbase Developer Portal which contains a ton of useful developer documentation.

The post Performance Enhancements to N1QL in the Couchbase 5.0 March Developer Build appeared first on The Couchbase Blog.

Categories: Architecture, Database

Get to know our Couchbase Community Meet ‚Äď Shivansh Srivastava

NorthScale Blog - 15 hours 10 min ago

In our latest interview for Get to Know Our Couchbase Community Champions, we meet Shivansh Srivastava from Delhi, India.

ShivanshSrivastava 181x300 1

Tell us about yourself and what you do in your daily role?

I consider myself to be a polyglot developer and a Scala, Spark, and Couchbase aficionado. I like to contribute to Open Source projects, and have contributed to projects like Apache iota, Apache Spark, Apache CarbonData, Couchbase-Spark-Connector, Akka, Alpakka, and many others. I’ve a keen interest in upcoming technologies like IoT, deep learning, etc. Currently, I’m employed as a Sr. Software Engineer at Chirpanywhere Inc, an IoT-based startup, where my main work is to manage many things from programming to designing the whole solution and deploying it. I’m also interested in building communities and am currently building my own community, Internity, to revolutionize the learning in colleges in India.

Where are you based and what do you like to do in your spare time?
I am currently based in Delhi, India. In my spare time I like to contribute to community building and help out others so they do not get stuck on the same problems I have experienced. Hence, I like to respond on Stackoverflow, write blogs, and learn from online courses (i.e., Coursera and edX, etc.). I also want to build a community offline, which is why I started the Internity initiative.

What made you choose Couchbase?
We’ve explored many options, but we chose Couchbase because of some pretty cool features like user-friendly UI, hence the cost of database management was reduced. It was in-memory, therefore, the reads were quite fast, and it has Couchbase Full Text Search (CBFT), which provides all the benefits of using a single database.

What one feature of Couchbase do you use and what should people know about it?

One feature is getting nearly ACID properties in a NoSQL database. Using the approach stated in Pro Couchbase Server by David Ostrovsky, Yaniv Rodenski, and Mohammed Haji, we formulated our solution and we were able to get nearly ACID properties in Couchbase. This book is a must-read.

What have you accomplished using Couchbase?

Earlier, I worked with a company related to the stock market and the data was semi-structured. With Couchbase, we were able to apply real-time N1QL queries to the data and integrate with Spark using the Spark-Couchbase Connector. And using Couchbase-Elastic river plugin to transfer data directly to ElasticSearch using the Couchbase UI is very user-friendly. There were many things we did not need to write code for that was managed by Couchbase itself, and the Couchbase UI has been very helpful in monitoring it as well.

If you could have three wishes for Couchbase what would they be?

My three wishes would be:

1. Increase the writing speed of the data which can be achieved in the near future.

2. Develop some compression algorithm like Pied Piper in Silicon Valley and inculcate into Couchbase to deal with the data volume.

3. To keep on increasing the open source activities to promote open source culture in other companies too.

What is your favorite book or author, and why?

My favourite book is Artificial Intelligence: A Modern Approach by Peter Norvig and Stuart J. Russell because of the ease with which they have explained such difficult topics. I also love to read biographies and autobiographies and am currently reading Playing It My Way, by Boria Majumdar and Sachin Tendulkar.

The post Get to know our Couchbase Community Meet – Shivansh Srivastava appeared first on The Couchbase Blog.

Categories: Architecture, Database

ANN: Couchbase Lite Developer Build 003 (Obj C, Swift, .NET)

NorthScale Blog - Fri, 03/24/2017 - 19:57

I am very pleased that Couchbase Lite Developer Build 003 is now available for download for Objective C, Swift and ‚Äú.NET‚ÄĚ.

Database queries have changed significantly in this release. Couchbase Lite 2.0 uses semantics based on N1QL query language defined by Couchbase. This API will be familiar If you’ve used Core Data, or other query APIs based on SQL (like jOOQ).

The Query API will have two styles – builder and fluent, and the implementation chosen would depend on the development platform.

Objective C

For Objective-C, we’ve implemented a builder API.

The SQL Statement

‚ÄėSELECT * FROM type=’account’ AND owner=’John’ ORDER BY dealSize‚Äô

can be written with the builder API as follows:

CBLQuery *query =

[CBLQuery select: [CBLQuerySelect all]

            from: [CBLQueryDataSource database: database]

¬†¬†¬†¬†¬† ¬†¬†¬†¬†where: [[CBLQueryExpression property: @”type”] equalTo: @”account”] and:

¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬†¬† [CBLQueryExpression property: @”owner”] equalTo: @”Wayne”]]

¬†¬†¬†¬†¬†¬†¬†¬† orderBy: [CBLQueryOrderBy expression: [CBLQueryExpression property: @”dealSize”]]

];

The API Specification for Objective C is available here and the release notes are here.

Swift

We’ve implemented a fluent API with support for the following

  • CRUD operations
  • Document with property type accessors
  • Subdocument API
  • Blob data type
  • CouchbaseLiteSwift framework for the Swift API
  • Database and Document Change Notification

The API Specification for Swift is available here and the release notes are here.

.NET

The

  • We‚Äôve added the sub document API which allows you to access embedded JSON objects. This is a much cleaner and safer approach with type safe accessors and the thread safety rules.
  • Thread safety checking is optional now. When used, it will be rigorously enforced and fail quickly to indicate incorrect usage.
  • Plus, you get everything that was available in DB002 which is listed here. The key highlights being:
    • A new and simplified API to interact with Couchbase Lite. CRUD operations are available and each subsequent build will have API expansion and changes. More details are available here.
    • Blob data type support.
    • Database and Document Change Notifications.

We are working hard on creating samples for you but in the meanwhile you can check out the unit tests here to get started.

The API specification is available here and the release notes are available here.

The post ANN: Couchbase Lite Developer Build 003 (Obj C, Swift, .NET) appeared first on The Couchbase Blog.

Categories: Architecture, Database

Working with Arrays in Couchbase with N1QL ‚Äď Video Tutorial

NorthScale Blog - Fri, 03/24/2017 - 17:00

In a recent article, I wrote about flattening and querying arrays in Couchbase using N1QL. ¬†That article was inspired by a popular question in the Couchbase Forums. ¬†After publishing the article, I felt a write-up alone couldn’t do it justice so I decided to create a video tutorial that walked through a bunch of queries on array data.

The video can be seen below:

I encourage you to read the write-up as well as it will help you when learning about N1QL and the cool things you can do with it.

For more information on using N1QL or Couchbase in general, check out the Couchbase Developer Portal.

The post Working with Arrays in Couchbase with N1QL – Video Tutorial appeared first on The Couchbase Blog.

Categories: Architecture, Database

Codenvy, Eclipse Che, Docker, and Couchbase: Coding in the Cloud at Devoxx US

NorthScale Blog - Thu, 03/23/2017 - 23:07

This week Couchbase was at Devoxx US.  Devoxx is a hugely influential conference series, well known in Europe.  This was the first one to take place in the United States.

While I was there I had a chance to talk with a group from Codenvy.  Codenvy allows you to quickly set up development stacks based on Docker containers.  They have several standard environments.  You can add to the existing ones or roll your own by adding Docker images.

Codenvy hosts everything in the cloud, so you can collaborate directly with other developers.  This means you can have applications that run against full-featured backend systems available at the click of a button.

I’m interested in using Codenvy for blogging. ¬†I like the idea of building a sample and having a link to code a reader can try out directly. ¬†So, I sat down during a break in the conference and knocked out a simple Couchbase Lite Java app. ¬†Codenvy has adopted Eclipse Che, which gives you a full featured editor that runs in your browser. ¬†(You can use any IDE and sync, too, if you don’t want to use Che.) ¬†I built the app using Che. ¬†The whole process was pretty easy. ¬†I’m looking forward to trying this out further, ideally with a complete Couchbase Mobile stack setup, including Couchbase Server running in its own container.

The app has no UI. ¬†It just shows how to open a database and perform simple CRUD operations. ¬†I used a helper class I find convenient that also wraps Couchbase replication, although I didn’t use that aspect.

I’ve barely scratched the surface on the features Codenvy offers. ¬†For now, click on the badge below and try it out with the sample I wrote.

Postscript

You can find more resources on our developer portal and follow us on Twitter @CouchbaseDev.

You can post questions on our forums. And we actively participate on Stack Overflow.

Hit me up on Twitter with any questions, comments, topics you’d like to see, etc. @HodGreeley

 

The post Codenvy, Eclipse Che, Docker, and Couchbase: Coding in the Cloud at Devoxx US appeared first on The Couchbase Blog.

Categories: Architecture, Database

FakeIt Series 3 of 5: Lean Models through Definitions

NorthScale Blog - Thu, 03/23/2017 - 20:05

Aaron Benton is an experienced architect who specializes in creative solutions to develop innovative mobile applications. He has over 10 years experience in full stack development, including ColdFusion, SQL, NoSQL, JavaScript, HTML, and CSS. Aaron is currently an Applications Architect for Shop.com in Greensboro, North Carolina and is a Couchbase Community Champion.

Aaron Benton

FakeIt Series 3 of 5: Lean Models through Definitions

In our previous post FakeIt Series 2 of 5: Shared Data and Dependencies we saw how to create multi-model dependencies with FakeIt. Today we are going to look at how we can create the same, but smaller models by leveraging definitions.

Definitions are a way of creating reusable set(s) within your model. It allows you to define a set of properties / values one time, referencing them multiple times throughout your model. Definitions in a FakeIt model are very similar to how definitions are used in the Swagger / Open API Specification.

Users Model

We will start with our users.yaml model that we defined in our first post.

name: Users
type: object
key: _id
properties:
  _id:
    type: string
    description: The document id built by the prefix "user_" and the users id
    data:
      post_build: `user_${this.user_id}`
  doc_type:
    type: string
    description: The document type
    data:
      value: user
  user_id:
    type: integer
    description: An auto-incrementing number
    data:
      build: document_index
  first_name:
    type: string
    description: The users first name
    data:
      build: faker.name.firstName()
  last_name:
    type: string
    description: The users last name
    data:
      build: faker.name.lastName()
  username:
    type: string
    description: The username
    data:
      build: faker.internet.userName()
  password:
    type: string
    description: The users password
    data:
      build: faker.internet.password()
  email_address:
    type: string
    description: The users email address
    data:
      build: faker.internet.email()
  created_on:
    type: integer
    description: An epoch time of when the user was created
    data:
      build: new Date(faker.date.past()).getTime()

Lets say that we have a new requirement where we have to support a home and work address for each user. Based on this requirement, we have decided to create a top-level property named addresses that will contain nested properties of home and work.

{
   ...
   "addresses": {
       "home": {
           "address_1": "123 Broadway St",
           "address_2": "Apt. C",
           "locality": "Greensboro",
           "region": "NC",
           "postal_code": "27409",
           "country": "US"
       },
       "work": {
           "address_1": "321 Morningside Ave",
           "address_2": "",
           "locality": "Greensboro",
           "region": "NC",
           "postal_code": "27409",
           "country": "US"
       }
   }
}

Our users.yaml model will need to be updated to support these new address properties.

(For brevity the other properties have been left off of the model definition)

...
properties:
 ...
  addresses:
    type: object
    description: An object containing the home and work addresses for the user
    properties:
      home:
        type: object
        description: The users home address
        properties:
          address_1:
            type: string
            description: The address 1
            data:
              build: `${faker.address.streetAddress()} ${faker.address.streetSuffix()}`
          address_2:
            type: string
            description: The address 2
            data:
              build: chance.bool({ likelihood: 35 }) ? faker.address.secondaryAddress() : null
          locality:
            type: string
            description: The city / locality
            data:
              build: faker.address.city()
          region:
            type: string
            description: The region / state / province
            data:
              build: faker.address.stateAbbr()
          postal_code:
            type: string
            description: The zip code / postal code
            data:
              build: faker.address.zipCode()
          country:
            type: string
            description: The country code
            data:
              build: faker.address.countryCode()
      work:
        type: object
        description: The users home address
        properties:
          address_1:
            type: string
            description: The address 1
            data:
              build: `${faker.address.streetAddress()} ${faker.address.streetSuffix()}`
          address_2:
            type: string
            description: The address 2
            data:
              build: chance.bool({ likelihood: 35 }) ? faker.address.secondaryAddress() : null
          locality:
            type: string
            description: The city / locality
            data:
              build: faker.address.city()
          region:
            type: string
            description: The region / state / province
            data:
              build: faker.address.stateAbbr()
          postal_code:
            type: string
            description: The zip code / postal code
            data:
              build: faker.address.zipCode()
          country:
            type: string
            description: The country code
            data:
              build: faker.address.countryCode()

As you can see our home and work address properties contain the exact same nested properties. This duplication makes our model bigger and more verbose. Additionally, what happens if our address requirements need to change? We’d have to make two updates to keep the structure the same. This is where we can take advantage of definitions, by defining a single way to create an address and reference it.

(For brevity the other properties have been left off of the model definition)

...
properties:
 ...
  addresses:
    type: object
    description: An object containing the home and work addresses for the user
    properties:
      home:
        description: The users home address
        schema:
         $ref: '#/definitions/Address'
      work:
        description: The users work address
        schema:
         $ref: '#/definitions/Address'
definitions:
  Address:
    type: object
    properties:
      address_1:
        type: string
        description: The address 1
        data:
          build: `${faker.address.streetAddress()} ${faker.address.streetSuffix()}`
      address_2:
        type: string
        description: The address 2
        data:
          build: chance.bool({ likelihood: 35 }) ? faker.address.secondaryAddress() : null
      locality:
        type: string
        description: The city / locality
        data:
          build: faker.address.city()
      region:
        type: string
        description: The region / state / province
        data:
          build: faker.address.stateAbbr()
      postal_code:
        type: string
        description: The zip code / postal code
        data:
          build: faker.address.zipCode()
      country:
        type: string
        description: The country code
        data:
          build: faker.address.countryCode()

Definitions are defined at the root of the model by specifying a definitions: property, then the name of the definition. Definitions are referenced by using $ref with the value being the path to the definition, for example #/definitions/Address. By using definitions we have saved almost 30 lines of code in our model and created a single place for how an Address is to be defined and generated within our Users model.

We can test the output of our updated Users model addresses using the following command:

fakeit console --count 1 models/users.yaml

blog3 example1

Now, lets say we have a requirement to store a Users main phone number, and then store optional additional phone numbers i.e. Home, Work, Mobile, Fax, etc. For this change we will use two new top-level properties: main_phone and additional_phones that is an array.

{
   ...
   "main_phone": {
       "phone_number": "7852322322",
       "extension": null
   },
   "additional_phones": [
       {
           "phone_number": "3368232032",
           "extension": "3233",
           "type": "Work"
       },
       {
           "phone_number": "4075922921",
           "extension": null,
           "type": "Mobile"
       }
   ]
}

While this may not be the most practical data model, or how I personally would have modeled this data we can use this example again to illustrate how we can take advantage of using definitions within our FakeIt model.

(For brevity the other properties have been left off of the model definition)

...
properties:
 ...
  main_phone:
    description: The users main phone number
    schema:
     $ref: '#/definitions/Phone'
    data:
      post_build: |
       delete this.main_phone.type
       return this.main_phone
  additional_phones:
    type: array
    description: The users additional phone numbers
    items:
     $ref: '#/definitions/Phone'
      data:
        min: 1
        max: 4
definitions:
  Phone:
    type: object
    properties:
      type:
        type: string
        description: The phone type
        data:
          build: faker.random.arrayElement([ 'Home', 'Work', 'Mobile', 'Other' ])
      phone_number:
        type: string
        description: The phone number
        data:
          build: faker.phone.phoneNumber().replace(/[^0-9]+/g, '')
      extension:
        type: string
        description: The phone extension
        data:
          build: chance.bool({ likelihood: 30 }) ? chance.integer({ min: 1000, max: 9999 }) : null

For this example we have created a Phone definition that contains 3 properties: type, phone_number and extension. You will notice that we have defined a post_build function on the main_phone property that removes the type attribute. This is illustrating how definitions can be used in conjunction with a build function to manipulate what is returned by the definition. The additional_phones property is an array of Phone definitions that will generate between 1 Р4 phones. We can test the output of our updated Users model phone numbers using the following command:

fakeit console --count 1 models/users.yaml

blog3 example2

Conclusion

Definitions allow you to streamline your FakeIt models by creating reusable sets of code for smaller more efficient models.

Up Next
  • FakeIt Series 4 of 5: Working with Existing Data
  • FakeIt Series 5 of 5: Rapid Mobile Development w/ Sync-Gateway
Previous

Couchbase Champion This post is part of the Couchbase Community Writing Program

The post FakeIt Series 3 of 5: Lean Models through Definitions appeared first on The Couchbase Blog.

Categories: Architecture, Database

TNDP Episode #16: The MEAN Stack for Application Development

NorthScale Blog - Thu, 03/23/2017 - 15:00

I am pleased to announce that the latest episode of The NoSQL Database Podcast titled, The Mean Stack for Application Development, has been published to all the popular podcasting networks.  In this episode I have special guest Jonathan Casarrubias from mean.expert come talk about the MongoDB, Express Framework, Angular, and Node.js (MEAN) development stack.

If you’re unfamiliar with the MEAN stack, it is a wildly popular development stack amongst JavaScript developers because all components in the stack have some relationship to JavaScript, making all the technologies play nice together.

This episode can be found on all the major podcast networks including iTunes and Pocket Casts, but if you’d prefer, you can listen to it below.

http://traffic.libsyn.com/nosql/TNDP_-_Episode_16_-_The_MEAN_Stack_for_Application_Development.mp3

If you’d like to follow up with Jonathan, you can find him on Twitter or via his website. ¬†If you’d like to learn about the previous podcast episode, find it here.

To learn more about Couchbase and what it has to offer, check out the documentation in the Couchbase Developer Portal.

The post TNDP Episode #16: The MEAN Stack for Application Development appeared first on The Couchbase Blog.

Categories: Architecture, Database

Hyper-V: How to run Ubuntu (or any Linux) on Windows

NorthScale Blog - Tue, 03/21/2017 - 15:57

I recently needed to install a Linux distro (Ubuntu being my favorite) in Hyper-V on Windows 10. I needed to do this because some internal builds of Couchbase were temporarily broken on Windows, but I still needed to use the latest builds. I’d never used Hyper-V to do this before, but it wasn’t that difficult. And the best part is that it comes with Windows: I don’t have to install VirtualBox or VMWare or anything like that.

Download Ubuntu

There are lots of great Linux distros out there, and Couchbase Server can be run on a variety of them. However, Ubuntu Desktop is my go-to.

The first thing I did was to download an Ubuntu Desktop ISO. I know that Couchbase support for Ubuntu 16 is coming, but for now I want to stick to Ubuntu 14. So, I downloaded an ISO of 14.04.5 (aka Trusty Tahr).

Make sure Hyper-V is enabled

Check to make sure that you have Hyper-V enabled on Windows.

First, open Control Panel. Next, go to Programs. Then, click “Turn Windows features on or off”. Finally, locate Hyper-V and click the checkbox (if it isn’t already checked).

Enable Hyper-V feature in Windows

You may also need to enable virtualization in the BIOS settings on your computer. The computer I’m using has it turned on by default. I wish I could give you instructions on how to do this, but it’s going to vary based on your BIOS.

Create a Hyper-V Virtual Switch

When connecting the virtual machine to a network, you’ll need to select a virtual switch. If you haven’t done this before, you’ll need to create one.

First, click “Virtual Switch Manager”, then select “External”. Click “Create Virtual Switch”. Finally, give the switch a name. (You also need to choose an external network; in my case I only have one). Hit “Apply”.

Create Hyper-V virtual switch

I wanted my machine to be connected to my home network, so I chose External. If you use External, it will connect to your network as a separate machine. After I create the machine (in the next section) it will appear on my router as it’s own machine:

Hyper-V machine attached to my network

Create a new Hyper-V Virtual Machine

Now you’re ready to create the virtual machine that will be running Ubuntu.

First, open the Hyper-V Manager from the Start menu. Then, click “New” and then “Virtual Machine”. This will start a wizard.

First, give your machine a name (like “UbuntuForCouchbase”). Optionally, you can choose where to store this machine, or use the default directory.

Next, you must choose Generation 1 or Generation 2. I used Generation 1, but Ubuntu 14+ should work on Generation 2 as well.

After that, specify how much memory to allocate to the machine. I have a lot of RAM on my desktop, so I gave it 8192mb.

Next, select the switch that was created in the previous section.

Then, setup the virtual hard disk to your liking. You can choose the location and size of the disk. I don’t need very much, so I went with 60gb.

Finally, you need to select an operating system. I’ll choose “Install an operating system from a bootable CD/DVD-ROM”, and then “Image file”, and find the Ubuntu ISO that I downloaded.

Click “Next”, view the summary, and click “Finish”.

Create Ubuntu Virtual Machine in Hyper-V

Start the machine and install Couchbase

Now that you’ve created a machine, you must start it. Right-click on the machine and select “Start” (or there’s a “Start” button in the actions menu to the right).

This starts the machine, but you can’t actually see the machine yet. You have to “Connect” to the machine to see what it’s doing. Right-click and select “Connect” (or again, there’s a “Connect” button in the actions menu to the right).

Ubuntu in Hyper-V

I won’t go through the installation process, since Ubuntu’s installation is pretty easy and self-explanatory (and well-documented).

At the end of the Ubuntu installation, installing Couchbase Server is also pretty easy and self-explanatory. Just go to the Couchbase downloads page and select the Ubuntu 14 version of Couchbase.

Download Couchbase for Ubuntu

Summary

I normally use the Windows version of Couchbase, but sometimes I need to use Ubuntu (or other Linux distros). Hyper-V on Windows is a convenient way for me to spin up virtual machines.

If you have any questions about Couchbase, please ask away in the Couchbase Forums or check out the Couchbase Developer Portal.

You can also contact me at Twitter @mgroves.

The post Hyper-V: How to run Ubuntu (or any Linux) on Windows appeared first on The Couchbase Blog.

Categories: Architecture, Database

Get to know our Couchbase Community Meet ‚Äď Abbas Ahmed

NorthScale Blog - Mon, 03/20/2017 - 10:18

Meet this week‚Äôs Couchbase Community Champion ‚Äď Abbas Ahmed from Rentalcars.com in Manchester!

abbas ahmed

Where are you based and what do you like to do in your spare time?

I am based in Manchester, U.K., which is also my birthplace and I like it a lot (discounting the weather). I am a dad of two beautiful kids and have a wonderful wife. The family gets a lot of my time, but if I manage to find any spare time it goes into my hobby of electronics. I’ve got an IoT network of Raspberry Pi’s, Arduinos and sensors at home that I like to add features to when I have a chance. I enjoy public speaking and have presented at Couchbase Manchester meetups and at Percona Live Amsterdam 2016. I also run regular internal training courses at Rentalcars.com.

Tell us about yourself and what you do in your daily role.

I got into computing through electronics, and started my career as a university lecturer with a great interest in databases and embedded systems. My first database was dBase III Plus, then I worked with FoxPro, MS Access, MS SQL Server and Oracle 6, 7 and 8. In 2000, during my first job after university, I started using MySQL and just fell in love with it. I had the opportunity to introduce MySQL cluster as a solution at my previous job in the Telco sector. I also had a chance to work with analytics data stores like InfiniDB and Infobright. On a typical day, I am working on planning and executing new projects for the Rentalcars.com Database Engineering team. I am mostly involved with application architects and make recommendations on how we can best implement database layers for new systems. Most of the new work is centered around microservices. We also have a DevOps culture, so there is a lot of focus on ensuring recurring tasks are automated.

What made you choose Couchbase?

Towards the end of 2013 Rentalcars.com was looking to add a new caching layer to some of our applications. We also had some upcoming JSON document database use cases that led us to study the available solutions. Couchbase offered us a cache without the cold cache problem, and it offered a decent document and K/V solution that had a clear roadmap for further improvement. Furthermore, the ease of management and effortless scaling of clusters were the decisive factors that made us choose Couchbase over the competition.

What have you accomplished using Couchbase?

We have managed to improve our user experience by a huge margin when it comes to page load times, and in terms of users not losing sessions when we decide to shift them from one data centre to another. We have also implemented caching using Couchbase to reduce the load on our supplier partners. We also have a couple of document use cases where we keep a massive collection of linked historical documents, so we have enjoyed brilliant availability with amazing performance characteristics. Couchbase Support have been stellar at making this all possible.

What one feature of Couchbase do you use and what should people know about it?

We use rack awareness (RA), which is a cool feature that ensures that no two copies of your data live in the same rack or blade chassis, hence guarding against data loss in case we lose a whole rack.

If you could have three wishes for Couchbase what would they be?

  1. Partial updates on XDCR.
  2. An official, supported NagiOS/statsd plugin to monitor all available Couchbase stats.
  3. Support for Graph structures (an official Titan add-on maybe).

What is your favorite book or author, and why?

Next Generation Databases by Guy Harrison is a book I read some time ago and I found it really well-written, with all the necessary concepts laid out in a pedagogical fashion. I don’t really have one favourite author, there are so many amazing people sharing their knowledge with us in the shape of books. For example: Deitel & Deitel, William Stallings, Andrew S. Tanenbaum, Albert Paul Malvino, and the list goes on.

Couchbase Champion

The post Get to know our Couchbase Community Meet – Abbas Ahmed appeared first on The Couchbase Blog.

Categories: Architecture, Database

N1QL: A Practical Guide

NorthScale Blog - Fri, 03/17/2017 - 23:02

N1QL is designed to help developer easily develop applications to solve real world problems.  Technically, N1QL is designed to give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. N1QL takes inspiration from SQL and extends SQL to handled flexible, nested structure of JSON.

This book is a compilation of articles on NoSQL concepts, N1QL features, optimization techniques, tools and use cases by N1QL R&D team, solution team from Couchbase.  We hope you use it, enjoy it and give feedback.

 A Practical Guide

N1QL: A Practical Guide

The post N1QL: A Practical Guide appeared first on The Couchbase Blog.

Categories: Architecture, Database

Authentication and Authorization with RBAC

NorthScale Blog - Fri, 03/17/2017 - 19:08

In March’s developer build, you can start to see some major changes to authentication and authorization within Role Based Access Control (RBAC) coming to Couchbase Server. These changes are a work in progress: the developer build is essentially a nightly build that gets released to the public. But there’s some good stuff in RBAC that’s worth getting excited about!

Go download the March 5.0.0 developer release of Couchbase Server today. Make sure to click the “Developer” tab to get the developer build (DB), and check it out. You still have time to give us some feedback before the official release.

Keep in mind that I’m writing this blog post on early builds, and some things may change in minor ways by the time you get the release, and some things may still be buggy.

Authentication and Authorization

Just a quick reminder of the difference between authentication and authorization:

  • Authentication is the process of identifying that a user is who they say they are.

  • Authorization is the process of making sure the user has permission to do what they are trying to do.

If you’ve used Couchbase before, you’re familiar with the login to what we sometimes call the “Admin Web Console”.

Couchbase authentication screen

However, the Web Console is really not just for admins, it’s for developers too. But until now, you didn’t really have a lot of control built-in to Couchbase about who can log in and (more importantly) what they’re allowed to do.

So, I’d like to introduce you to Couchbase’s new first-class user feature.

Users

There’s still a full administrator user. This is the login that you create when you first install Couchbase. This is the user who is unrestricted, and can do anything, including creating new users. So, for instance, a full administrator can see the “Security” link in the navigation, while other users can’t.

Security link to manage authentication and authorization

Now, once on this security page, you can add, edit, and delete users.

A user can identify a person, but it can also identify some service or process. For instance, if you’re writing an ASP.NET application, you may want to create a user with a limited set of permissions called “web-service”. Therefore, the credentials for that “user” would not be for a person, but for an ASP.NET application.

Next, try adding a new Couchbase user by clicking “+ Add User”. I’m going to create a user called “fts_admin”, with a name of “Full Text Search Admin”, a password, and a single role: FTS Admin of the travel-sample bucket (FTS = Full Text Search).

Adding a new User

Here’s an animation of adding that user:

Add a new user with Couchbase authentication

Some notes about the above animation:

  • I selected “Couchbase” instead of “External”. External is meant for LDAP integration. Note that “Couchbase” (internal authentication) will likely become the default in future releases.

  • FTS Admin gives the user permission to do everything with Full Text Searches: create, modify, delete, and execute them.

  • I granted FTS Admin only for the travel-sample bucket. If I selected “all”, that would grant permission to all buckets, even ones created in the future.

  • Users with the FTS Searcher role only have access to execute searches, not modify or create them.

More on the difference between FTS Admin and FTS Searcher later.

Logging in as a new user

Now that this user is created, I can login as fts_admin. This user’s authentication is handled within Couchbase.

Login with Couchbase authentication

First, in the above animation, note that the fts_admin user has a much more limited set of options compared to the full admin user.

Next, it’s worth pointing out that users can reset their password:

Reset password

Creating an FTS index

Since I’ve already created an fts_admin user with the FTS Admin role, I’ll create another user called fts_searcher that only has the FTS Searcher role for the travel-sample bucket.

List of users

Using the REST API for FTS

I’m going to use the REST API to demonstrate that these users are limited by the roles I’ve given them. If you need a refresher on the REST API, you can refer to the documentation of the Full Text Search API. Also note that I’m using the REST API because there are some bugs in the UI as I’m writing this.

Let’s start by creating a new Full Text Search (FTS) index. I’ll do this via Postman, but you can use curl or Fiddler or whatever REST tool you prefer.

Create an FTS index

To create an index with the REST API, I need to make a PUT request to the /api/index/<indexname> endpoint.

  • First, I’ll create an index for the ‘hotel’ type in the travel-sample bucket, so I’ll PUT to /api/index/hotels

  • Also, credentials can be put in the URL to use basic authentication

  • Furthermore, the REST endpoints are available on port 8094

Finally, the URL for the PUT request should look something like this:

http://fts_searcher:password@192.168.1.10:8094/api/index/hotels

The body of the PUT is a big JSON object. Below is part of it. You can find the full version on GitHub to try for yourself.

{
  "type": "fulltext-index",
  "name": "hotels",
  "sourceType": "couchbase",
  "sourceName": "travel-sample",

// ... snip ...

}

Normally, you can create this via the UI instead of having to create JSON by hand. I’m not going to go into FTS in much detail in this post, because my goal is to demonstrate the new authentication and authorization features, not FTS itself.

Trying to create an index without authorization

Notice that I’m using fts_searcher as the user. I know that fts_searcher shouldn’t have permission to create indexes, so I would expect a 403. And that’s just what I get.

{
  "message": "Forbidden. User needs one of the following permissions",
  "permissions": [
    "cluster.bucket[travel-sample].fts!write"
  ]
}

So, while the authentication worked, that user doesn’t have the necessary authorization.

Creating an index with authorization

I’ll try again with fts_admin:

http://fts_admin:password@192.168.1.10:8094/api/index/hotels

And assuming an index named ‘hotels’ doesn’t already exist, you’ll get a 200, and this in the body of response:

{
  "status": "ok"
}
Using the FTS index

Next, let’s use the REST API to search the index for the word ‘breakfast’.

First, make a POST to the /api/index/hotels/query endpoint, again with the proper credentials and port number.

http://fts_admin:password@192.168.1.10:8094/api/index/hotels/query

or

http://fts_searcher:password@192.168.1.10:8094/api/index/hotels/query

Both users should be able to execute a search using that index.

Next, in the body of the POST should be a simple JSON object. Again, you don’t normally have to create this by hand — your SDK of choice or the Web Console UI can do this for you.

{
  "explain": true,
  "fields": [
    "*"
  ],
  "highlight": {},
  "query": {
    "query": "breakfast"
  }
}

Finally, the result of this search request will be a large JSON response. Look within the “hits” sub-document for “fragments” to verify that the search worked. Here’s a snippet of my search for “breakfast”. Again, the full result is on Github.

// ... snip ...

        "reviews.content": [
          "√Ę‚ā¨¬¶ to watch TV. <mark>Breakfast</mark> was served every morning along with a copy of the Times-Picayune. I took my <mark>breakfast</mark> downstairs in the patio, the coffee was very good. The continental <mark>breakfast</mark> is nothing to√Ę‚ā¨¬¶"
        ]
      },

// ... snip ...
This is a preview, expect some bugs!

There are some bugs and some incomplete features.

  • I’ve shown FTS roles here on purpose. This is because the other roles are not yet fully formed. Please try them out, let us know what you think, but remember they are not in their final form. FTS is closest to ready.

  • I’ve seen some issues when logging in as a non-admin user causes the web console to behave badly. Because of this, I showed the REST example above instead of relying on the UI.

  • Finally, there might be other bugs that we don’t know about yet. Please let us know! You can file an issue in our JIRA system at issues.couchbase.com or submit a question on the Couchbase Forums. Or, contact me with a description of the issue. I would be happy to help you or submit the bug for you (my Couchbase handlers send me a cake pop when I submit a good bug).

If you have questions, the best way to contact me is either Twitter @mgroves or email me matthew.groves@couchbase.com.

The post Authentication and Authorization with RBAC appeared first on The Couchbase Blog.

Categories: Architecture, Database

Feature Enhancements to N1QL in the Couchbase Server 5.0 March Developer Build

NorthScale Blog - Fri, 03/17/2017 - 16:50

With the recent March 2017 Developer build of Couchbase there were many bug fixes, but also feature enhancements to core technologies like N1QL.  For example, now indexes can be created on various meta information such as the document cas and expiration values.  Another example includes a simplified syntax when creating array indexes.

We’re going to take a look at some of these enhancements and how you might use them in your own application.

Simplified Array Indexing in Couchbase 5.0

With Couchbase Server 4.5 came the array indexing. Take the following sample document for example:

{
  "type": "person",
  "firstname": "Nic",
  "lastname": "Raboy",
  "social-media": [
    {
      "type": "twitter",
      "url": "https://www.twitter.com/nraboy"
    },
    {
      "type": "website",
      "url": "https://www.thepolyglotdeveloper.com"
    }
  ]
}

New Array Index Syntax

Before Couchbase 5.0, to index the array elements found in social-media you had to write an index that looked something like the following:

CREATE INDEX ism 
ON `default` ( DISTINCT ARRAY media FOR media IN `social-media` END )
WHERE type = 'person';

In the above example, the FOR operator was necessary for array indexing.  In Couchbase Server 5.0 there is a much more simplified syntax.  The same index can be created via the following:

CREATE INDEX ism 
ON `default` ( DISTINCT `social-media` )
WHERE type = "person";

To make sure your index works, you can execute the following N1QL query:

EXPLAIN SELECT *
FROM default
WHERE type = 'person' AND ANY media IN `social-media` SATISFIES media.type = 'website' END;

When looking through the results of the EXPLAIN you should see that it is using the ism index that was previously created.

Now just because the simplified syntax exists doesn’t mean you can’t use the previous syntax when array indexing. ¬†The following would be a perfect example of why the previous syntax would still be valid:

CREATE INDEX ism_website
ON `default` ( DISTINCT ARRAY media FOR media IN `social-media` WHEN media.type = 'website' END )
WHERE type = 'person';

Notice that a WHEN operator was used when creating the ism_website index above.

More information on array indexing can be found here, along with other useful documentation on creating indexes in Couchbase.

Relaxed Variable Match Requirement for Array Indexes

In 4.x releases, the array indexing required usage of exactly same variable names in the SELECT query that were used in the CREATE INDEX statement. Find more details here.

For example, referring to the previous queries seen above, note that the variable media that is used to iterate through the array social-media is very important. Array indexing in 4.x mandates the exact variable name media to be used in the previous SELECT query.

Couchbase 5.0 release relaxes this requirement, and following query would work perfectly fine:

EXPLAIN SELECT *
FROM default
USE INDEX (ism)
WHERE type = 'person' AND ANY m IN `social-media` SATISFIES m.type = 'website' END;

Note that, the index ism uses variable name media, but the query above uses m. Still, the above query can use the index ism successfully.

A Relaxed Whole Array Index Key Requirement for Array Indexes

Also recall, in 4.x releases, the covered array indexing requires the whole array attribute as a mandatory index-key in the array index definition.  For example, take the following query again:

EXPLAIN SELECT *
FROM default
WHERE type = 'person' AND ANY media IN `social-media` SATISFIES media.type = 'website' END;

The covered array index corresponding to the above query would be:

CREATE INDEX ism_covered 
ON `default` ( DISTINCT ARRAY media FOR media IN `social-media` END,  `social-media`)
WHERE type = 'person';

Note that, the second index key social-media is mandatory, for example to cover following query:

EXPLAIN SELECT media
FROM default
WHERE type = 'person' AND ANY media IN `social-media` SATISFIES media IS NOT NULL END;

In Couchbase 5.0, this same query will be covered by index ism at the very start of this article.

It is important to note that, this feature brings lot of power to Array Indexes. Because, now each entry in the array index consumes less storage space and memory. It enables following benefits:

  • Usage of covered array indexes for larger arrays
  • Brings more efficiency and performance to queries using covered array indexes
Indexing Document Meta Information

Let’s take a turn here and discuss the new meta information that can be indexed. ¬†Previously indexes could be created on the meta().id property, but now both the meta().cas and meta().expiration properties are supported.

So how do we create an index that uses the meta properties as keys? ¬†It isn’t any different from what you already know. ¬†Take the following covering indexes for example:

CREATE INDEX idx_cas
ON `default` ( META().cas, META().expiration )

CREATE INDEX idx_exp
ON `default` ( META().expiration )

Now if I wanted to use the idx_cas and idx_exp indexes in a N1QL query, I could do something like the following:

SELECT META().id, META().cas
FROM default
WHERE META().cas > 1489531586248179712;

SELECT META().id, META().expiration
FROM default
WHERE META().expiration > NOW_MILLIS();

So why would being able to index these properties?  Well, what if you wanted to do a query for all documents that had expired today or all documents that were modified today?

For more information on the CAS and expiration properties, visit here.  For more information on indexing or using N1QL with Couchbase, check out the Couchbase Developer Portal.

The post Feature Enhancements to N1QL in the Couchbase Server 5.0 March Developer Build appeared first on The Couchbase Blog.

Categories: Architecture, Database

Profiling and Monitoring in Couchbase Server 5.0 Preview (Update)

NorthScale Blog - Thu, 03/16/2017 - 20:06

In March’s developer build, there are some more updates for N1QL query monitoring and profiling.

Go download the March 5.0.0 developer release of Couchbase Server today. Make sure to click the “Developer” tab to get the developer build (DB), and check it out. You still have time to give us some feedback before the official release.

In case you missed it, check out the post I wrote in February about the New Profiling and Monitoring in Couchbase Server 5.0 Preview, as this post very much builds on that.

As before, keep in mind that I’m writing this blog post on early builds, and some things may change in minor ways by the time you get the release.

Query Workbench

Once again, I‚Äôll be focusing on Query Workbench for this blog post. The updates for March are mainly visual changes to the “Plan” view in Query Results.

But just to review, there are other options for running N1QL queries:

Personally, I find the Query Workbench easiest to use, as it more visually presents the profiling.

Profiling complex queries

Let’s look at the travel-sample data again, just like I did in last month’s post. I’m using the travel-sample bucket, but I have removed one of the indexes (DROP INDEX travel-sample.def_sourceairport;).

I then execute a N1QL query to find routes between two cities. Let’s use Columbus, Ohio and Denver, Colorado this time.

SELECT r.id, a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment
FROM `travel-sample` r
UNNEST r.schedule s
JOIN `travel-sample` a ON KEYS r.airlineid
WHERE r.sourceairport = 'CMH'
AND r.destinationairport = 'DEN'
AND s.day = 0
ORDER BY a.name;

Executing this query (on my single-node local machine) took about 8 seconds this time (as expected), which is too slow.

Visual Breakdown of Profiling

Let’s look at the plan to see what the problem might be (I broke it into two lines so the screenshots will fit in the blog post).

Visual profiling part 1

Visual profiling part 2

So, as before, the costliest parts of the query plan are the Filter and the Join. We could tell before by looking at the raw numbers and/or the percentages. But in this March release, we have a more visual way to tell: color. The parts of the plan go from gray to tan to gold based on percentages and defined thresholds.

Right now, the thresholds are based on the fraction of the total query time taken by an operation:

  • Gray: less than 1% of total time
  • Tan/Some gold: √Ę‚ā¨‚Äú 1% – 5%
  • Tan/More gold: 5% – 20%
  • All gold: 20%

The purpose of this visual profiling is to quickly draw your eye to expensive operations. Then, if you care to know the exact numbers, you can read it in the details (in the diagram or even in the META().plan information).

I’m not going to go through the index creation part again; it’s the same as it was in the last blog post (just recreating the index that I removed for demonstration purposes).

We still want your feedback!

Stay tuned to the Couchbase Blog for information about what’s coming in the next developer build.

Interested in trying out some of these new features? Download Couchbase Server 5.0 today!

We want feedback! Developer releases are coming every month, so you have a chance to make a difference in what we are building.

Do you like this new use of color to help you profile your queries? Now you can give feedback directly from within the Couchbase Web Console. Look for the feedback icon] icon at the bottom right of the screen.

Send feedback from within Couchbase

Is something not working right? Please file an issue in our JIRA system at issues.couchbase.com or submit a question on the Couchbase Forums. Or, contact me with a description of the issue. I would be happy to help you or submit the bug for you (my Couchbase handlers let me have a free Kit-Kat for each good bug I submit).

If you have questions, the best way to contact me is either Twitter @mgroves or email me matthew.groves@couchbase.com.

The post Profiling and Monitoring in Couchbase Server 5.0 Preview (Update) appeared first on The Couchbase Blog.

Categories: Architecture, Database

FakeIt Series 2 of 5: Shared Data and Dependencies

NorthScale Blog - Thu, 03/16/2017 - 14:58

Aaron Benton is an experienced architect who specializes in creative solutions to develop innovative mobile applications. He has over 10 years experience in full stack development, including ColdFusion, SQL, NoSQL, JavaScript, HTML, and CSS. Aaron is currently an Applications Architect for Shop.com in Greensboro, North Carolina and is a Couchbase Community Champion.

Aaron Benton

FakeIt Series 2 of 5: Shared Data and Dependencies

In FakeIt Series 1 of 5: Generating Fake Data we learned that FakeIt can generate a large amount of random data based off a single YAML file and output the results to various formats and destination, including Couchbase Server. Today we are going to explore what makes FakeIt truly unique and powerful in the world of data generation.

There are tons of random data generators available, a simple Google Search will give you more than enough to choose from. However, almost all of these have the same frustrating flaw, which is they can only ever deal with a single model. Rarely as developers do we have the luxury of dealing with a single model, more often than not we are developing against multiple models for our projects. This is where FakeIt stands out, it allows for multiple models and those models to have dependencies.

Let’s take a look at the possible models we’ll have within our e-commerce application:

  • Users
  • Products
  • Cart
  • Orders
  • Reviews

Users, the first model that we defined does not have any dependencies and the same can be said for the Products model, which we will define next. However, it would be logical to say that our Orders model would depend on both the Users and Products model. If we truly want test data, the documents created by our Orders model should be the actual random data generated from both the Users and Products models.

Products Model

Before we look at how model dependencies work in FakeIt let’s define what our Products model is going to look like.

name: Products
type: object
key: _id
properties:
  _id:
    type: string
    description: The document id
    data:
      post_build: `product_${this.product_id}`
  doc_type:
    type: string
    description: The document type
    data:
      value: product
  product_id:
    type: string
    description: Unique identifier representing a specific product
    data:
      build: faker.random.uuid()
  price:
    type: double
    description: The product price
    data:
      build: chance.floating({ min: 0, max: 150, fixed: 2 })
  sale_price:
    type: double
    description: The product price
    data:
      post_build: |
       let sale_price = 0;
       if (chance.bool({ likelihood: 30 })) {
         sale_price = chance.floating({ min: 0, max: this.price * chance.floating({ min: 0, max: 0.99, fixed: 2 }), fixed: 2 });
       }
       return sale_price;
  display_name:
    type: string
    description: Display name of product.
    data:
      build: faker.commerce.productName()
  short_description:
    type: string
    description: Description of product.
    data:
      build: faker.lorem.paragraphs(1)
  long_description:
    type: string
    description: Description of product.
    data:
      build: faker.lorem.paragraphs(5)
  keywords:
    type: array
    description: An array of keywords
    items:
      type: string
      data:
        min: 0
        max: 10
        build: faker.random.word()
  availability:
    type: string
    description: The availability status of the product
    data:
      build: |
       let availability = 'In-Stock';
       if (chance.bool({ likelihood: 40 })) {
         availability = faker.random.arrayElement([ 'Preorder', 'Out of Stock', 'Discontinued' ]);
       }
       return availability;
  availability_date:
    type: integer
    description: An epoch time of when the product is available
    data:
      build: faker.date.recent()
      post_build: new Date(this.availability_date).getTime()
  product_slug:
    type: string
    description: The URL friendly version of the product name
    data:
      post_build: faker.helpers.slugify(this.display_name).toLowerCase()
  category:
    type: string
    description: Category for the Product
    data:
      build: faker.commerce.department()
  category_slug:
    type: string
    description: The URL friendly version of the category name
    data:
      post_build: faker.helpers.slugify(this.category).toLowerCase()
  image:
    type: string
    description: Image URL representing the product.
    data:
      build: faker.image.image()
  alternate_images:
    type: array
    description: An array of alternate images for the product
    items:
      type: string
      data:
        min: 0
        max: 4
        build: faker.image.image()

This model is a little more complex than our previous Users model. Let’s examine a few of this property in more detail:

  • _id: This value is being set after every property in the document has been build and is available to the post build function. The this¬†context is that of the current document being generated
  • sale_price: This using defining a 30% chance of a sale price and if there is a sale price ensuring that the value is less than that of the price¬†property
  • keywords: Is an array. This defined similarly to Swagger, we define our array items and how we want them constructed using the build¬†/ post_build¬†functions. Additionally, we can define min¬†and max¬†values and FakeIt will generate a random number of array elements between these values. There is also a fixed¬†property that can be used to generate a set number of array elements.

Now that we’ve constructed our Products model let’s generate some random data and output it to the console to see what it looks like using the command:

fakeit console models/products.yaml

blog2 example1 1

Orders Model

For our project we have already defined the following models:

  • users.yaml
  • products.yaml

Let’s start by defining or Orders model without any properties and specifying its dependencies:

name: Orders
type: object
key: _id
data:
  dependencies:
    - products.yaml
    - users.yaml
properties:

We have defined two dependencies for our Orders model, and referenced them by their file name. Since all of our models are stored in the same directory there is no reason to specify the full path. At runtime, FakeIt will first parse all of the models before attempting to generate documents, and it will determine a run order based on each of the models dependencies (if any).

Each of the build functions in a FakeIt model is a function body, with the following arguments passed to it.

function (documents, globals, inputs, faker, chance, document_index, require) {
 return faker.internet.userName();
}

Once the run order has been established, each of the dependencies are saved in-memory and made available to the dependant model through the documents argument. This argument is an object containing a key for each model whose value is an array of each document that has been generated. For our example of the documents property it will look similar to this:

{
 "Users": [
   ...
 ],
 "Products": [
   ...
 ]
}

We can take advantage of this to retrieve random Product and User documents assigning their properties to properties within our Orders model. For example, we can retrieve a random user_id from the documents generated by the Users model and assign that to the user_id of the Orders model through a build function

user_id:
    type: integer
    description: The user_id that placed the order
    data:
      build: faker.random.arrayElement(documents.Users).user_id;

Let’s define what the¬†rest of our¬†Orders model will look like:

name: Orders
type: object
key: _id
data:
  dependencies:
    - products.yaml
    - users.yaml
properties:
  _id:
    type: string
    description: The document id
    data:
      post_build: `order_${this.order_id}`
  doc_type:
    type: string
    description: The document type
    data:
      value: "order"
  order_id:
    type: integer
    description: The order_id
    data:
      build: document_index + 1
  user_id:
    type: integer
    description: The user_id that placed the order
    data:
      build: faker.random.arrayElement(documents.Users).user_id;
  order_date:
    type: integer
    description: An epoch time of when the order was placed
    data:
      build: new Date(faker.date.past()).getTime()
  order_status:
    type: string
    description: The status of the order
    data:
      build: faker.random.arrayElement([ 'Pending', 'Processing', 'Cancelled', 'Shipped' ])
  billing_name:
    type: string
    description: The name of the person the order is to be billed to
    data:
      build: `${faker.name.firstName()} ${faker.name.lastName()}`
  billing_phone:
    type: string
    description: The billing phone
    data:
      build: faker.phone.phoneNumber().replace(/x[0-9]+$/, '')
  billing_email:
    type: string
    description: The billing email
    data:
      build: faker.internet.email()
  billing_address_1:
    type: string
    description: The billing address 1
    data:
      build: `${faker.address.streetAddress()} ${faker.address.streetSuffix()}`
  billing_address_2:
    type: string
    description: The billing address 2
    data:
      build: chance.bool({ likelihood: 50 }) ? faker.address.secondaryAddress() : null
  billing_locality:
    type: string
    description: The billing city
    data:
      build: faker.address.city()
  billing_region:
    type: string
    description: The billing region, city, province
    data:
      build: faker.address.stateAbbr()
  billing_postal_code:
    type: string
    description: The billing zip code / postal code
    data:
      build: faker.address.zipCode()
  billing_country:
    type: string
    description: The billing region, city, province
    data:
      value: US
  shipping_name:
    type: string
    description: The name of the person the order is to be billed to
    data:
      build: `${faker.name.firstName()} ${faker.name.lastName()}`
  shipping_address_1:
    type: string
    description: The shipping address 1
    data:
      build: `${faker.address.streetAddress()} ${faker.address.streetSuffix()}`
  shipping_address_2:
    type: string
    description: The shipping address 2
    data:
      build: chance.bool({ likelihood: 50 }) ? faker.address.secondaryAddress() : null
  shipping_locality:
    type: string
    description: The shipping city
    data:
      build: faker.address.city()
  shipping_region:
    type: string
    description: The shipping region, city, province
    data:
      build: faker.address.stateAbbr()
  shipping_postal_code:
    type: string
    description: The shipping zip code / postal code
    data:
      build: faker.address.zipCode()
  shipping_country:
    type: string
    description: The shipping region, city, province
    data:
      value: US
  shipping_method:
    type: string
    description: The shipping method
    data:
      build: faker.random.arrayElement([ 'USPS', 'UPS Standard', 'UPS Ground', 'UPS 2nd Day Air', 'UPS Next Day Air', 'FedEx Ground', 'FedEx 2Day Air', 'FedEx Standard Overnight' ]);
  shipping_total:
    type: double
    description: The shipping total
    data:
      build: chance.dollar({ min: 10, max: 50 }).slice(1)
  tax:
    type: double
    description: The tax total
    data:
      build: chance.dollar({ min: 2, max: 10 }).slice(1)
  line_items:
    type: array
    description: The products that were ordered
    items:
      type: string
      data:
        min: 1
        max: 5
        build: |
         const random = faker.random.arrayElement(documents.Products);
         const product = {
            product_id: random.product_id,
            display_name: random.display_name,
            short_description: random.short_description,
            image: random.image,
            price: random.sale_price || random.price,
            qty: faker.random.number({ min: 1, max: 5 }),
         };
         product.sub_total = product.qty * product.price;
         return product;
  grand_total:
    type: double
    description: The grand total of the order
    data:
      post_build: |
       let total = this.tax + this.shipping_total;
       for (let i = 0; i < this.line_items.length; i++) {
         total += this.line_items[i].sub_total;
       }
       return chance.dollar({ min: total, max: total }).slice(1);

And output it to the console using the command:

fakeit console models/orders.yaml

blog2 example2 1

As you can see from the console output, the documents were generated for the Users and Products models, and those documents were made available to the Orders model. However, they were excluded from output because all that was requested to be output was the Orders model.

Now that we have defined 3 models with dependencies (Users, Products and Orders), we need to be able to generate multiple documents for each of these and output them to Couchbase Server. Up to this point we have been specifying the number of documents to generate via the –count¬†command line argument. We can specify the number of documents or a range of documents by using the data:¬†property at the root of the model.

users.yaml

name: Users
type: object
key: _id
data:
  min: 1000
  max: 2000

products.yaml

name: Products
type: object
key: _id
data:
  min: 4000
  max: 5000

orders.yaml

name: Orders
type: object
key: _id
data:
  dependencies:
    - products.yaml
    - users.yaml
  min: 5000
  max: 6000

We can now generate random sets of related document models and output those documents directly into Couchbase Server using the command:

fakeit couchbase --server 127.0.0.1 --bucket ecommerce --verbose models/

blog2 example3 1

Conclusion

We’ve seen through three simple FakeIt YAML models how we can create model dependencies allowing for randomly generated data to be related across models and streamed into Couchbase Server. We’ve also seen how we can specify the number of documents to generate by model by using the data:¬†property at the root of a model.

These models can be stored in your projects repository, taking up very little space and allow your developers to generate the same data structures with completely different data. Another advantage of being able to generate documents through multi-model relationships is to explore different document models and see how they perform with various N1QL queries.

Up Next
  • FakeIt Series 3 of 5: Lean Models through Definitions
  • FakeIt Series 4 of 5: Working with Existing Data
  • FakeIt Series 5 of 5: Rapid Mobile Development w/ Sync-Gateway
Previous

Couchbase Champion This post is part of the Couchbase Community Writing Program

The post FakeIt Series 2 of 5: Shared Data and Dependencies appeared first on The Couchbase Blog.

Categories: Architecture, Database

Couchbase 5.0 March 2017 Developer Build Features and Enhancements

NorthScale Blog - Wed, 03/15/2017 - 21:02

It is that time again, another month, another developer build for Couchbase.  This time we have a ton of bug fixes and feature enhancements as we get one step closer to the stable release of Couchbase 5.0.

The March 2017 developer build can be downloaded from the Couchbase downloads page in the developer tab.

Bug Fixes

Here are some bugs we fixed in the March Developer Build, all of which can be found in our Jira issue tracker.

MB-23142 IN clause is not using IndexCountScan MB-23126 cbimport says “-g” option is optional, but in fact it is required MB-23109 Incorrect StoredValue size calculations MB-23106 panic found in indexer – plasma ubuntu longevity run MB-23097 Disable DISTINCT pushdown for Joins/NEST/UNNEST MB-23090 [FTS] document the store configuration JSON, especially w/ mossStore config MB-23082 cluster stuck post rebalance in plasma longevity MB-23073 with index2 api enabled encoded_plan doesn’t work properly MB-23059 Reset size of byte slice from buffer pool to 0 for Distinct scans MB-23047 [Index API2] Distinct=true gives wrong results when projection is passed as nil MB-23029 Query failure with Panic: runtime error – invalid memory address or nil pointer dereference MB-23022 Spock-UI: Cannot tweak internal settings MB-23018 avoid calling getThroughSeqnos and getHighSeqnoAndVBUuidFromTarget more than once MB-22999 UI: no checkbox to confirm failover in new UI MB-22992 [N1QL] Explain plan still shows (meta(default).expiry) though index was created on meta().expiration MB-22988 [FTS] upgrade to go 1.8 MB-22982 Tree form query output fails to print in query workbench MB-22975 Couchbase-cli user-manage help is incorrect for auth-type MB-22974 Creating user in CLI requires mandatory full name, but creating user through UI accepts optional full name MB-22969 With long username the RBAC UI becomes unusable since the buttons for edit/delete vanish MB-22962 [FTS] performance: use 6 pindexes by default instead of 32 MB-22954 Radio buttons for failover options is not visible in the new UI MB-22942 Create audit trail for privilege debug MB-22937 [FTS] UI: too much white-space around index mapping checkboxes MB-22935 [FTS] UI: copy-to-clipboard curl doesn’t handle escaping single quotes correctly MB-22931 [FTS] UI: API created index definition has fts.js JSON parse problem MB-22925 Rebalance fails with “Client.Timeout exceeded while awaiting headers” MB-22898 Web UI does not have security tab on Mac MB-22897 Indexer crashes when we do start/stop/start indexer rebalance in a loop MB-22892 couchbase-cli wants -c/–cluster but man page reads: “by default this…” MB-22891 For Dynamic index IndexProjection is empty when it uses covered index MB-22890 Query Workbench – periodic failures due to panic in cbq-engine MB-22885 Create index causes UI to hang MB-22884 PAIRS() not returning all pairs MB-22882 Index never gets created in MDS setup MB-22880 N1QL query panel becomes inaccessible when indexer rebalance is going on MB-22861 Replica indexes do not show up on new UI MB-22860 cbq_engine panics when creating universal index MB-22859 [Dynamic Index] “Index out of range stack” panic when creating an index MB-22858 [Dynamic Index] panic and timeout when processing multi-predicates query MB-22856 View index replicas parameter should not be accepted when creating and ephemeral bucket MB-22854 [N1QL] Panic seen in query.log: panic: runtime error: index out of range [recovered] MB-22853 [Index API2] Distinct as hint to Scan2() eliminates more items MB-22852 10-15% drop in Q1-Q3 query throughput MB-22851 [FTS] UI: checkbox labels need to be clickable MB-22848 Rebalance fails after a failed cbindex move MB-22847 Query engine exit creating index via workbench MB-22846 [Dynamic Indexing] Panic observed in query when dynamic index is being created on a bucket with nested document MB-22844 Fix n1ql race conditions MB-22843 Spock UI: Error messages get duplicated in UI MB-22836 [FTS] perf: initial indexing rate seems suboptimal MB-22831 N1QL : Inserting a numerical value from N1QL causes the value to be stored in science notation MB-22830 couchbase-cli crashed when using option server-readd –help MB-22828 [Index API2] len(IndexProjection.EntryKeys) == 0 should not project the keys MB-22825 Typo of heartbeats to hearbeats MB-22819 Merge and update queries failing with Error “DML Error, possible causes include CAS mismatch or concurrent modification Failed to perform update” MB-22817 Fix covering of META MB-22808 [Dynamic Indexing] SUM related query failed in case of dynamic indexing MB-22793 [FTS] queries should work when using the ngram and edge_ngram analyzers MB-22787 Upgrade xdcr to golang 1.7.5 MB-22778 Getl with no extra fields receives “Key Not found” response where on older versions it locks for default time. MB-22774 cli messages for unauthorized users is regressed MB-22771 error when running – admin-role-manage with switch –get-roles MB-22768 /checkPermissions – not returning 401 for non-existent user in cb server MB-22760 unable to create buckets using /pools/default/buckets API MB-22757 Handle mix of covering and non-covering scans MB-22755 new UI doesn’t redirect to the welcome page if node is removed from its cluster MB-22747 Ephemeral buckets: After delete curr_items is not decremented MB-22734 View index may be crashed MB-22711 [FTS] moss ReadOnly options should open file in read-only mode MB-22705 Missing items with Swap Rebalance MB-22677 [cbq] generic index fails under load (err: backfill exceeded limit 5120) MB-22656 upon activation n1ql node thinks it’s not part of a cluster MB-22652 query against system:indexes does not show all indexes when run from QWB MB-22650 Indexer does not recover if it is in DGM MB-22633 cbq-engine and indexer are getting killed MB-22632 [Query] Query metrics – timings should be EE only MB-22614 If the number of replicas is changed on a bucket the user should be told a rebalance is required MB-22588 N1QL clustering missing n1ql nodes MB-22536 DATE_RANGE_STR() and DATE_RANGE_MILLIS() give wrong results MB-22527 [Spock UI] – Can’t copy document metadata MB-22461 [FTS] allow more moss configurability from index definition JSON MB-22447 UI – Full Text Search should be promoted to spock’s left-hand-side menu MB-22437 [Query Workbench UI] : Issues with Query Editor Preferences dialog MB-22421 XATTR CAS Macro is incorrect MB-22416 [FTS] QueryString syntax doesn’t have short way to perform exact numeric equality search MB-22310 [Spock UI] The ‘Activity’ link/tab on the UI is unresponsive MB-22273 plans set via encoded_plan REST parameter may return wrong results MB-22268 Spock UI :”Servers Failed Over” in Dashboard is left aligned instead of center MB-22260 Update our favicon.ico MB-22197 Query UI caches list of valid nodes too aggressively MB-22087 N1QL: Incorrect results for queries using intersect scan as compared to same query using primary index MB-22069 Spock UI : Number of nodes involved in the rebalance is not show in the UI. MB-22060 subquery expression returns error it panics MB-21988 JOIN with expression in from clause panics MB-21944 [FTS] perf: date facets query doesn’t scale due to extensive garbage collection (top CPU load is 35%) MB-21862 [Spock UI] FTS – index definition is incorrectly stripped of type mapping with empty name (“”) MB-21857 [Spock UI] : FTS – UI scrambles when the Target Full Text Indexes list / Index name is very long MB-21843 Spock UI : n1ql query link should point to new link MB-21643 [FTS] feature to show the pindex given a docId (support / diagnostic feature) MB-21452 [FTS] perf: term + date facets query is very slow MB-21441 [FTS] FTS cmd-line tools need warnings MB-21280 Request sent for every line in cbq’s batch mode MB-21215 Please don’t echo query when using “cbq -q” MB-21191 [N1QL] ‘msg\’: u\’FROM expression term should not have USE KEYS\’ MB-20999 [Spock UI] : Add id/class/name attributes to UI elements in the new UI MB-20992 [FTS] query string query with only -termanalyzedtonothing different results from ES MB-20583 [FTS] UI: Enable index replication via FTS UI MB-20311 [FTS] REST: Validate index definition if char_filter and token_filter is passed as an array MB-20042 substr should start from 1

For more information in regards to developing with Couchbase, check out the Couchbase Developer Portal.

The post Couchbase 5.0 March 2017 Developer Build Features and Enhancements appeared first on The Couchbase Blog.

Categories: Architecture, Database

FakeIt Series 1 of 5: Generating Fake Data

NorthScale Blog - Tue, 03/14/2017 - 22:35

Aaron Benton is an experienced architect who specializes in creative solutions to develop innovative mobile applications. He has over 10 years experience in full stack development, including ColdFusion, SQL, NoSQL, JavaScript, HTML, and CSS. Aaron is currently an Applications Architect for Shop.com in Greensboro, North Carolina and is a Couchbase Community Champion.

Aaron Benton

FakeIt Series 1 of 5: Generating Fake Data

There are countless blog posts on data modeling, key and document patterns. All of these posts give a great introduction into how to structure and model your documents in Couchbase, but none of them tell you what to do next. In this blog series we are going to answer the question, what do you after you’ve defined your data model?

Users Model

For this series we will be working with a greenfield e-commerce application. As with most e-commerce applications, our application is going to have users so this is where we will begin.

We have defined a basic user model to start with.

{
 "_id": "user_0",
 "doc_type": "user",
 "user_id": 0,
 "first_name": "Mac",
 "last_name": "Carter",
 "username": "Salma.Ratke",
 "password": "DvA6YrMGtgsKKnG",
 "email_address": "Ludie74@hotmail.com",
 "created_on": 1457172796088
}

We’ve done the hardest part, which is defining our model, but now what?

  • How do we represent this model?
  • How do we document this model?
  • Does this model rely on data from other models?
  • How can data be generated from this model?
  • How can we generate fake / test data?

Luckily for us there is a NodeJS project called FakeIt that can answer all of these questions for us. FakeIt is a command-line utility that generates fake data in json, yaml, yml, cson, or csv formats based on models which are defined in yaml. Data can be generated using any combination of FakerJS, ChanceJS, or Custom Functions. The generated data can be output in the following formats and destinations:

  • json
  • yaml
  • cson
  • csv
  • Zip Archive of json, yaml, cson or csv files
  • Couchbase Server
  • Couchbase Sync Gateway Server

We can define a FakeIt model in YAML to represent our JSON model. This provides us a documented and data-typed model that we can communicate how our model should be structure and what the properties are for.

name: Users
type: object
key: _id
properties:
  _id:
    type: string
    description: The document id built by the prefixed "user_" and the users id
  doc_type:
    type: string
    description: The document type
  user_id:
    type: integer
    description: The users id
  first_name:
    type: string
    description: The users first name
  last_name:
    type: string
    description: The users last name
  username:
    type: string
    description: The users username
  password:
    type: string
    description: The users password
  email_address:
    type: string
    description: The users email address
  created_on:
    type: integer
    description: An epoch time of when the user was created

You’re probably saying to yourself, ‚Äúgreat, I’ve defined my model in YAML but what good does this do me?‚ÄĚ One of the biggest issues developers face when beginning development is having data to work with. Often times an exorbitant amount of time is spent manually creating documents, writing throw away code to populate a bucket. Additionally you may have a full or partial data dump of your database that has to be imported.

These are time consuming, tedious and in the case of a data dump do not provide any insight or documentation into the available models. We can add a few simple properties to our FakeIt model describing how our model should be generated, and through a single file we can create an endless amount of fake randomized documents.

name: Users
type: object
key: _id
properties:
  _id:
    type: string
    description: The document id built by the prefix "user_" and the users id
    data:
      post_build: `user_${this.user_id}`
  doc_type:
    type: string
    description: The document type
    data:
      value: user
  user_id:
    type: integer
    description: An auto-incrementing number
    data:
      build: document_index
  first_name:
    type: string
    description: The users first name
    data:
      build: faker.name.firstName()
  last_name:
    type: string
    description: The users last name
    data:
      build: faker.name.lastName()
  username:
    type: string
    description: The username
    data:
      build: faker.internet.userName()
  password:
    type: string
    description: The users password
    data:
      build: faker.internet.password()
  email_address:
    type: string
    description: The users email address
    data:
      build: faker.internet.email()
  created_on:
    type: integer
    description: An epoch time of when the user was created
    data:
      build: new Date(faker.date.past()).getTime()

We have added a data property to each of our models properties describing how that value should be generated. FakeIt supports 5 different ways to generate a value:

  • pre_build:¬†function to initialize the value
  • build:¬†function that builds a value
  • fake:¬†A FakerJS template string i.e. {{internet.userName}}
  • value:¬†A static value to use
  • post_build:¬†a function that runs after every property in the model has been set

These build functions are a JavaScript function body. Each of these functions is passed the following variables that can be used at the time of its execution:

  • documents¬†– An object containing a key for each model whose value is an array of each document that has been generated
  • globals¬†– An object containing any global variables that may have been set by any of the run or build functions
  • inputs¬†– An object containing a key for each input file used whose value is the deserialized version of the files data
  • faker¬†– A reference to FakerJS
  • chance¬†– A reference to ChanceJS
  • document_index¬†– This is a number that represents the currently generated document’s position in the run order
  • require¬†– This is the node require function, it allows you to require your own packages. For better performance require and seth them in the pre_run function.

For example, if we look at the username properties build function it would look like this:

function (documents, globals, inputs, faker, chance, document_index, require) {
 return faker.internet.userName();
}

Now that we have defined how our model should be generated, we can start to generate some fake data with it.

With our users model saved to a file models/users.yaml, we can output data directly to the console using the command

fakeit console models/users.yaml

blog1 example1

Using this same model we can generate 100 JSON files and save them into a directory named output/ using the command

fakeit directory ‚Äďcount 100 ‚Äďverbose output models/users.yaml

blog1 example2

Additionally, we can create a zip archive of 1,000 JSON files using the command:

fakeit directory ‚Äďcount 1000 ‚Äďverbose output/users.zip models/users.yaml

blog1 example3

We can even generate a single CSV file of our model using the following command:

fakeit directory ‚Äďcount 25 ‚Äďformat csv ‚Äďverbose output/ models/users.yaml

This will create a single CSV file whose name is the name of the model, in this case name: Users with the resulting file being named Users.csv

blog1 example4

Whether you are using JSON files, Zip Archives or CSV files all of these can be imported into Couchbase Server by using the CLI tools cbdocloader (for *.json and *.zip files) or cbimport (for *.json and *.csv files)

While generating static files is beneficial, there is still the extra step of having to import them into Couchbase Server through the available CLI tools. FakeIt also supports Couchbase Server and Sync Gateway as output destinations. We can generate 10,000 JSON documents from our users.yaml model, and output them to a bucket named ecommerce on a Couchbase Server running locally using the command:

fakeit couchbase ‚Äďserver 127.0.0.1 ‚Äďbucket ecommerce ‚Äďcount 10000 ‚Äďverbose models/users.yaml

blog1 example5

Conclusion

We’ve seen how we can represent a user’s JSON model using YAML to document and describe how a properties value should be generated. That single users.yaml¬†file can be output to the console, JSON files, Zip archive of JSON files, CSV files, and even directly into Couchbase. FakeIt is a fantastic tool to speed up your development and generate larger development datasets. You can save your FakeIt models as part of your codebase for easy repeatable datasets by any developer.

FakeIt is a tool to ease development and testing of your Couchbase deployment. While it can generate large amounts of data, it is not a true load testing tool. There are CLI tools available for load testing and sizing such as cbc-pillowfight and cbworkloadgen

Up Next
  • FakeIt Series 2 of 5: Shared Data and Dependencies
  • FakeIt Series 3 of 5: Lean Models through Definitions
  • FakeIt Series 4 of 5: Working with Existing Data
  • FakeIt Series 5 of 5: Rapid Mobile Development w/ Sync-Gateway

Couchbase Champion

This post is part of the Couchbase Community Writing Program

The post FakeIt Series 1 of 5: Generating Fake Data appeared first on The Couchbase Blog.

Categories: Architecture, Database

NDP Episode #15: Cassandra and the Tabular Database

NorthScale Blog - Tue, 03/14/2017 - 18:49

I am pleased to announce that the latest episode of The NoSQL Database Podcast titled, Cassandra and the Tabular Database, has been published to all the major podcasting networks.  In this episode, special guest speaker Tim Berglund and I discuss Cassandra and how it fits as a NoSQL database solution.

In case you’re unfamiliar, Tim Berglund is a VP at DataStax and a repeat¬†guest on the NoSQL Database Podcast. ¬†The last time he was on the show was when we had discussed NoSQL in general on the very first episode. ¬†He is a great guy and very knowledgable on Cassandra and DataStax.

Episode #15 can be downloaded from iTunes, Pocket Casts, and many other networks. ¬†However, if you’d like to listen to it here, you can find it below.

http://traffic.libsyn.com/nosql/TNDP_-_Episode_15_-_Cassandra_and_the_Tabular_Database.mp3

If you’re interested in hearing the previous episode that Tim was a guest on, it can be found here.

To learn more about NoSQL and Couchbase in particular, check out the Couchbase Developer Portal for information on querying, architecture, and using any of the many developer SDKs.

The post NDP Episode #15: Cassandra and the Tabular Database appeared first on The Couchbase Blog.

Categories: Architecture, Database

Get to know our Couchbase Community Meet ‚Äď Ian Merrington

NorthScale Blog - Tue, 03/14/2017 - 08:03

In this week’s Get to Know Our Couchbase Community Champion, we go to the Sky! Meet Ian Merrington from London who works at Sky U.K.

ian merrington

Tell us about yourself and what you do in your daily role.

I work in the OTT Persistence engineering team for Sky. Sky has really embraced the NoSQL concept, and the Persistence Engineering team is responsible for making sure it‚Äôs done right across all the teams. Day to day I‚Äôm creating the tooling required to successfully manage Couchbase, from Ansible deployment scripts to Grafana monitoring. I‚Äôm also the escalation point for any Couchbase issues across Sky, and work with teams both pre and post ‚Äúgo live‚ÄĚ to ensure they are making full use of the product. In my previous life I was an Oracle DBA for 18 years, but who wants to hear about that.

Where are you based and what do you like to do in your spare time?

I am based in Sky’s Osterley Campus in London, so rubbing shoulders with Football players, F1 Drivers and Game of Thrones White Walkers on a daily basis. Outside of work I’m a huge music fan, so when not annoying the neighbors with my terrible guitar playing I will be at one of my local venues, pint glass in hand.

What made you choose Couchbase?

When I was working on the Sky Identity platform we had a requirement to migrate the platform to another data center. XDCR was the perfect tool to do this, so we migrated the application to Couchbase, switched on XDCR, and there we were, in our new datacenter. Coming from an Oracle background the simplicity of the replication configuration converted me to Couchbase.

What have you accomplished using Couchbase?

Couchbase allowed us to hit all our growth targets for the Identity platform, with fantastic resiliency. The introduction of N1QL allowed us to finally switch off the Reporting RDBMS that had been underpinning the whole platform.

What one feature of Couchbase do you use and what should people know about it?

I‚Äôm an old school DBA ‚Äď so for me the introduction of the audit¬†log and user roles bring Couchbase into line with RDBMS standards. Make sure your audit log is being ingested into somewhere you can query it now, rather than when you really need it. And make sure your users only have the privileges they need, rather than putting this into place once someone has accidently deleted something from the live system.

If you could have three wishes for Couchbase what would they be?

Instead of having metric windows of 1min, 1hour, and 1day, can we please have something between 1 min and 1 hour? The number of times I have been asked, ‚Äúwhat happened‚ÄĚ and I get to their desk just as the issue has moved out the 1minute window and isn‚Äôt visible in the 1hour window is too often. It would also be good to have monitoring plugins for a few of the popular metric collection engines, though I have enjoyed building my own.

What is your favorite book or author, and why?

I’m going to be greedy and choose a favorite author and a favorite book. Author would have to be Douglas Adams. Yeah, I’m a geek, but his ability to pull together so many sci-fi concepts into stories that are hilarious as well as thought provoking is an art in itself. Plus, he was a huge environmentalist as well as a Doctor Who script writer, so what’s not to love? My favorite book is Microserfs by Douglas Coupland (It seems I have never made it out of the Douglas section at my local bookstore). Whilst it’s written with the 90s tech bubble as a backdrop, the book really is about the friendships and comradery you can find in any workplace.

 

Couchbase Champion

The post Get to know our Couchbase Community Meet – Ian Merrington appeared first on The Couchbase Blog.

Categories: Architecture, Database

Offline-first apps with Couchbase Sync Gateway

NorthScale Blog - Mon, 03/13/2017 - 17:17

Rafael Ugolini

Rafael Ugolini is a full stack software developer currently working at Famoco in Brussels, Belgium. He has been working with software development for more than 10 years and lately he is focused on designing web solutions and developing using Python and Javascript. Twitter, LinkedIn, Github

Introduction

In this article, I’m going to propose an offline-first development using Couchbase as the communication tool between the backend and frontend.

While studying react-redux, something that I really liked is the Async Actions pattern where you have:

{ type: 'FECH_POSTS_REQUEST' }

{ type: 'FECH_POSTS_FAILURE', error: 'Oops' }

{ type: 'FECH_POSTS_SUCCESS', response: { ... } }

After understanding a bit more about Couchbase and Couchbase Sync Gateway, it seems possible to apply this kind of pattern and let Couchbase deal with all the communication of the App and having a fully functional offline app.This pattern provides a very good user experience because you actually render the page based on the current status of the app.

How does it work?

How does it work

Application

The application itself never does any call to the backend, the only responsibility it has is to save states and render them, this way it’s possible to have a fully functional app working offline.

Couchbase Lite/Sync Gateway

Couchbase Lite will be responsible to sync the current state of the App to Couchbase Sync Gateway and retrieve new information once the document is updated in the backend.

Web hooks

Once Couchbase Sync Gateway receives a document that matches the filter, it will do an HTTP call to the Web App with the document it needs to update.

Web App

The Web App can do any kind of update like:

  • Creating a new Task in a Task Queue
  • Retrieving data from an external API
  • Analyze some data/images (eg: OCR)
Hands On

To illustrate a bit how this concept works, I develop a simple example that once you save a specific document, the web application will dispatch a task that will get a random Star Wars character and update the document.

You can download all the code necessary to run an example of this solution.

The stack is composed by:

  • Web App – Flask
  • Task Queue – Celery +¬†RabbitMQ
  • Sync Gateway (walrus mode)

To have it running, simple clone the git repo and run docker-compose:

git clone https://github.com/rafaelugolini/syncgateway_apiless_example

cd syncgateway_apiless_example

docker-compose up

Data Modeling

In this example, the key action is the primary source of the events, the document must be saved with:

{ "action": "person_request" }

To create a document curl, just use the following command

curl -H "Content-Type: application/json" -X POST -d '{"action":"person_request"}' http://localhost:4984/db/

Sync Gateway

In the configuration of the sync gateway, its registered an event handler that every document change with action == ‚Äúperson_request‚ÄĚ, an API call will be made to the web app.

"event_handlers": {

"document_changed": [

{"handler": "webhook",

"url": "http://web_service:5000/person_request/",

"filter": `function(doc) {

if (doc.action == "person_request") {

return true;

}

return false;

}`

}

]

}

Web App

The web app is a simple Flask API that receives a POST with the information of the document and dispatches a task to Celery.

The task will query a random Star Wars person from https://swapi.co/ and update the document:

{

"action": "person_success",

"gender": "female",

"height": "168",

"mass": "55",

"name": "Zam Wesell"

}

 

Client

I developed a simple pouchdb client that will print all the documents from sync gateway to the console. It basically runs this function every time there is a database change.

const getAllDocs = () => (
  db.allDocs({
    include_docs: true,
    attachments: true,
  }).then((result) => {
    console.log('\x1Bc'); // this clears the console
    console.log(util.inspect(result.rows, false, null));
  }).catch((err) => {
    console.log(err);
  })
);

You can get it from this git repo.

yarn install
yarn start

 

This post was brought to you from the Couchbase Community Writing Program

 

The post Offline-first apps with Couchbase Sync Gateway appeared first on The Couchbase Blog.

Categories: Architecture, Database