Tag Archives: NoSQL

Installing and Running MongoDB from Scratch

I’m relatively new to Mongo, but I’ve decided to use it for a recent project, mainly because it writes quickly. I wanted to install and run it locally; however, most of the tutorials for Mongo push Atlas pretty heavily, so I thought it might be worth creating a post on installing and running Mongo from scratch. Most of what’s here is a duplication of what you can find in the official docs.

Download

Start by downloading Mongo from here.

You’re then given some options during the installation. If you decide to change any of the default directories, then make a note of what you change them to; otherwise just accept the defaults:

This process will install MongoDb Compass by default – this is Mongo’s equivalent of SQL Server Management Studio, or MySQL’s SQL Workbench.

Running the Service

Mongo will install a Windows Service by default (well, if you look closely at the screenshot above, you’ll see that it’s pretty easy to stop it doing this), and you can just connect to this; however, you can remove that service (just go into Windows -> Services and remove it).

If you choose to keep the default installation then skip down to the Compass step.

In order to run Mongo yourself, you’ll need to run a command window, and navigate to the Mongo installation directory (typically C:\Program Files\MongoDB\Server\4.4\bin):

The key files here are:
– mongo.exe which is a mongo client
– mongod.exe which is the database service

The first thing to do is to start the DB Service:

.\mongod.exe –dbpath=”c:\tmp\mongodata”

This will use c:\tmp to store the data in.

Running the Client and Manipulating the Data

Now that the server is running, you can launch the client:

First, launch a command prompt as admin.

Next navigate to: C:\Program Files\MongoDB\Server\4.4\bin\

PS C:\Program Files\MongoDB\Server\4.4\bin> .\mongo.exe

This is a command line client. To create a new DB, just tell it to use a database that doesn’t exist:

> use testdb
switched to db testdb
> db.testcollection.insertOne({test: "test", test2: 1})
{
        "acknowledged" : true,
        "insertedId" : ObjectId("60830bfe34473a83b0fe56a6")
}
>

Here, we’ve inserted an object into the database; we can view that object by using something like the following:

> db.testcollection.find()
{ "_id" : ObjectId("60830bfe34473a83b0fe56a6"), "test" : "test", "test2" : 1 }
>

Finally, we can do the same thing, but in a GUI, called Compass.

Compass

After launching Compass for the first time, select New Connection, and then Fill in connection fields individually:

The following will work if you’ve just installed the default (as above) and made no config changes:

We can now browse the data that we inserted earlier:

References

https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/

Google Cloud Datastore – Setting up a new Datastore and accessing it from a console application

Datastore is a NoSql offering from Google. It’s part of their Google Cloud Platform (GCP). The big mind shift, if you’re used to a relational database is to remember that each row (although they aren’t really rows) in a table (they aren’t really tables) can be different. The best way I could think about it was a text document; each line can have a different number of words, numbers and symbols.

However, just because it isn’t relational, doesn’t mean you don’t have to consider the structure; in fact, it actually seems to mean that there is more onus on the designer to consider what and where the data will be used.

Pre-requisites

In order to follow this post, you’ll need an account on GCP, and a Cloud Platform Project.

Set-up a New Cloud Datastore

The first thing to do is to set-up a new Datastore:

Zones

The next step is to select a Zone. The big thing to consider, in terms of cost and speed is to co-locate your data where possible. Specifically with data, you’ll incur egress charges (that is, you’ll be charged as your data leaves its zone), so your zone should be nearby, and co-located with anything that accesses it. Obviously, in this example, you’re accessing the data from where your machine is located, so pick a zone that is close to where you live.

In Britain, we’re in Europe-West-2:

Entities and Properties

The next thing is to set-up new entity. As we said, an entity is loosely analogous to a table.

Now we have an entity, the entity needs some properties. This, again, is loosely analogous to a field; if the fields were not required to be consistent throughout the table. I’m unsure how this works behind the scenes, but it appears to simply null out the columns that have no value; I suspect this may be a visual display thing.

You can set the value (as above), and then query the data, either in a table format (as below):

Or, you can use a SQL like syntax (as below).

Credentials

In order to access the datastore from outside the GCP, you’ll need a credentials file. You;ll need to start off in the Credentials screen:

In this instance, we’ll set-up a service account key:

This creates the key as a json file:

The file should looks broadly like this:

{
  "type": "service_account",
  "project_id": "my-project-id",
  "private_key_id": "private_key_id",
  "private_key": "-----BEGIN PRIVATE KEY-----\nkeydata\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "clientid",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-project-id%40appspot.gserviceaccount.com"
}

Keep hold of this file, as you’ll need it later.

Client Library

There is a .Net client library provided for accessing this functionality from your website or desktop app. What we’ll do next is access that entity from a console application. The obvious first step is to create one:

Credentials again

Remember that credentials file I said to hang on to; well now you need it. It needs to be accessible from your application; there’s a number of ways to address this problem, and the one that I’m demonstrating here is probably not a sensible solution in real life, but for the purpose of testing, it works fine.

Copy the credentials file into your project directory and include it in the project, then, set the properties to:

Build Action: None
Copy to Output Directory: Copy if Newer

GCP Client Package

You’ll need to install the correct NuGet package:

Install-Package Google.Cloud.Datastore.V1

Your Project ID

As you use the GCP more, you’ll come to appreciate that the project ID is very important. You’ll need to make a note of it (if you can’t find it, simply select Home from the hamburger menu):

The Code

All the pieces are now in place, so let’s write some code to access the datastore:

Environment.SetEnvironmentVariable(
    "GOOGLE_APPLICATION_CREDENTIALS",
    Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "my-credentials-file.json"));
 
GrpcEnvironment.SetLogger(new ConsoleLogger());
 
// Your Google Cloud Platform project ID
string projectId = "my-project-id";
 
DatastoreClient datastoreClient = DatastoreClient.Create();
 
DatastoreDb db = DatastoreDb.Create(projectId, "TestNamespace", datastoreClient);

string kind = "MyTest";

string name = "newentitytest3";
KeyFactory keyFactory = db.CreateKeyFactory(kind);
Key key = keyFactory.CreateKey(name);
 
var task = new Entity
{
    Key = key,
    ["test1"] = "Hello, World",
    ["test2"] = "Goodbye, World",
    ["new field"] = "test"
};
 
using (DatastoreTransaction transaction = db.BeginTransaction())
{                
    transaction.Upsert(task);
    transaction.Commit();
}

If you now check, you should see that your Datastore has been updated:

There’s a few things to note here; the first is that you will need to select the right Namespace and Kind. Namespace defaults to [default], and so you won’t see your new records until you select that.

When things go wrong

The above instructions are deceptively simple; however, getting this example working was, by no means, straight-forward. Fortunately, when you have a problem with GCP and you ask on StackOverflow, you get answered by Jon Skeet. The following is a summary of an error that I encountered.

System.InvalidOperationException

System.InvalidOperationException: ‘The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.’

The error occurred on the BeginTransaction line.

The ConsoleLogger above isn’t just there for show, and does give some additional information; in this case:

D1120 17:59:00.519509 Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library “C:\Users\pmichaels.nuget\packages\grpc.core\1.4.0\lib\netstandard1.5../..\runtimes/win/native\grpc_csharp_ext.x64.dll” D1120 17:59:00.600298 Grpc.Core.Internal.NativeExtension gRPC native library loaded successfully. E1120 17:59:02.176461 0 C:\jenkins\workspace\gRPC_build_artifacts\platform\windows\workspace_csharp_ext_windows_x64\src\core\lib\security\credentials\plugin\plugin_credentials.c:74: Getting metadata from plugin failed with error: Exception occured in metadata credentials plugin.

It turns out that the code was failing somewhere in here. Finally, with much help, I managed to track the error down to being a firewall restriction.

References

https://cloud.google.com/datastore/docs/reference/libraries

https://cloud.google.com/datastore/docs/concepts/entities?hl=en_US&_ga=2.55488619.-171635733.1510158034

https://cloud.google.com/dotnet/

https://github.com/GoogleCloudPlatform/dotnet-docs-samples

https://developers.google.com/identity/protocols/application-default-credentials

https://cloud.google.com/datastore/docs/concepts/overview

https://cloud.google.com/datastore/docs/reference/libraries