Tag Archives: servicestack

Partial Updates with HTTP PATCH using ServiceStack.net and the JSON Patch format (RFC 6902)

I have been looking into implementing partial updates using the HTTP PATCH method using ServiceStack.net and the JSON Patch format (RFC 6902)

This is of interest since many updates do not neatly match the PUT method, which often is used for full entity updates (all properties). PATCH is intended to do one or more partial updates. There are a few blogs describing the use cases.

I’ve been happy using ServiceStack the way it was designed – RESTful, simple, using Message Based designs.

I could implement PATCH using my own message format – that is easy to do. Usually it would be the actual DTO properties, plus a list of fields which are actually going to be updated. You wouldn’t update all fields, and you don’t want to only update non-null properties, since sometimes “null” is a valid value for a property (it would be impossible to set a property to null from non-null).

In my opinion, using JSON Patch for the Request body has pros and cons.
Pros:

  • is an official RFC
  • covers a lot of use cases

Cons:

  • very generic, so we lose some of the benefit of strong typing
  • doesn’t have a slot for the Id of a resource when calling PATCH /employees/{Id}
    • doing this the “JSON Patch way” would be { “op”: “replace”, “path”: “/employees/123/title”, “value”: “Administrative Assistant” } , but that wastes the value of having it on the routing path.

JSON Patch supports a handful of operations: “add”, “remove”, “replace”, “move”, “copy”, “test”. I will focus on the simple “replace” op, since it easily maps to replacing a property on a DTO (or field in a table record).

The canonical example looks like this:

PATCH /my/data HTTP/1.1
Host: example.org
Content-Length: 55
Content-Type: application/json-patch+json
If-Match: "abc123"

[
    { "op": "replace", "path": "/a/b/c", "value": 42 }
]

I’m going to ignore the If-Match: / ETag: headers for now. Those will be useful if you want to tell the server to only apply your changes if the resource still matches your “If-Match” header (no changes in the meantime). “That exercise is left to the reader.”

Let’s say we have a more practical example:

  • an Employee class, backed by an [Employee] table, accessed by OrmLite
  • an EmployeeService class, implementing the PATCH method
  • the Request DTO to the Patch() method aligns to the JSON Patch structure

The Employee class would simply look like this (with routing for basic CRUD):

[Route("/employees", "GET,POST")]
[Route("/employees/{Id}", "GET,PUT")]
public class Employee
{
    public long Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
    public string Title { get; set; }
    public int? CubicleNo { get; set; }
    public DateTime StartDate { get; set; }
    public float Longitude { get; set; }
    public float Latitude { get; set; }
}

Now the shape of JSON Patch replace ops would look like this:

PATCH /employees/123 HTTP/1.1
Host: example.org
Content-Type: application/json

[
    { "op": "replace", "path": "/title", "value": "Junior Developer" },
    { "op": "replace", "path": "/cubicleno", "value": 23 },
    { "op": "replace", "path": "/startdate", "value": "2013-06-02T09:34:29-04:00" }
]

The path is the property name in this case, and the value is what to update to.

And yes, i also know i am sending Content-Type: application/json instead of Content-Type: application/json-patch+json . We’ll have to get into custom content type support later too.

Now, sending a generic data structure as the Request DTO to a specific resource ID doesn’t cleanly map to the ServiceStack style, because:

  • each Request DTO should be a unique class and route
  • there is not a field in the Request for the ID of the entity

The simple way to map the JSON to a C# class would define an “op” element class, and have a List<T> of them, like so:

public class JsonPatchElement
{
    public string op { get; set; } // "add", "remove", "replace", "move", "copy" or "test"
    public string path { get; set; }
    public string value { get; set; }
}

We create a unique Request DTO so we can route to the Patch() service method.

[Route("/employees/{Id}", "PATCH")]
public class EmployeePatch : List<JsonPatchElement>
{
}

But how do we get the #$%&& Id from the route?? This code throws RequestBindingException! But i can’t change the shape of the PATCH request body from a JSON array [].
The answer was staring me in the face: just add it to the DTO class definition, and ServiceStack will map to it. I was forgetting the C# class doesn’t have to be the same shape as the JSON.

[Route("/employees/{Id}", "PATCH")]
public class EmployeePatch : List<JsonPatchElement>
{
    public long Id { get; set; }
}

Think of this class as a List<T> with an additional Id property.

When the method is called, the JSON Patch array is mapped and the Id is copied from the route {Id}.

public object Patch(EmployeePatch dto)
{
    // dto.Id == 123
    // dto[0].path == "/title"
    // dto[0].value == "Joe"
    // dto[1].path == "/cubicleno"
    // dto[1].value == "23"

The only wrinkle is all the JSON values come in as C# string, even if they are numeric or Date types. At least you will know the strong typing from your C# class, so you know what to convert to.

My full Patch() method is below– note the partial update code uses reflection to update properties of the same name, and does primitive type checking for parsing the string values from the request DTO.

public object Patch(EmployeePatch dto)
{
    // partial updates

    // get from persistent data store by id from routing path
    var emp = Repository.GetById(dto.Id);

    if (emp != null)
    {
        // read from request dto properties
        var properties = emp.GetType().GetProperties();

        // update values which are specified to update only
        foreach (var op in dto)
        {
            string fieldName = op.path.Replace("/", "").ToLower(); // assume leading /slash only for example

            // patch field is in type
            if (properties.ToList().Where(x => x.Name.ToLower() == fieldName).Count() > 0)
            {
                var persistentProperty = properties.ToList().Where(x => x.Name.ToLower() == fieldName).First();

                // update property on persistent object
                // i'm sure this can be improved, but you get the idea...
                if (persistentProperty.PropertyType == typeof(string))
                {
                    persistentProperty.SetValue(emp, op.value, null);
                }
                else if (persistentProperty.PropertyType == typeof(int))
                {
                    int valInt = 0;
                    if (Int32.TryParse(op.value, out valInt))
                    {
                        persistentProperty.SetValue(emp, valInt, null);
                    }
                }
                else if (persistentProperty.PropertyType == typeof(int?))
                {
                    int valInt = 0;
                    if (op.value == null)
                    {
                        persistentProperty.SetValue(emp, null, null);
                    }
                    else if (Int32.TryParse(op.value, out valInt))
                    {
                        persistentProperty.SetValue(emp, valInt, null);
                    }
                }
                else if (persistentProperty.PropertyType == typeof(DateTime))
                {
                    DateTime valDt = default(DateTime);
                    if (DateTime.TryParse(op.value, out valDt))
                    {
                        persistentProperty.SetValue(emp, valDt, null);
                    }
                }

            }
        }

        // update
        Repository.Store(emp);

    }

    // return HTTP Code and Location: header for the new resource
    // 204 No Content; The request was processed successfully, but no response body is needed.
    return new HttpResult()
    {
        StatusCode = HttpStatusCode.NoContent,
        Location = base.Request.AbsoluteUri,
        Headers = {
            // allow jquery ajax in firefox to read the Location header - CORS
            { "Access-Control-Expose-Headers", "Location" },
        }
    };
}

For an example of calling this from the strongly-typed ServiceStack rest client, my integration test looks like this:

[Fact]
public void Test_PATCH_PASS()
{
    var restClient = new JsonServiceClient(serviceUrl);

    // dummy data
    var newemp1 = new Employee()
    {
        Id = 123,
        Name = "Kimo",
        StartDate = new DateTime(2015, 7, 2),
        CubicleNo = 4234,
        Email = "test1@example.com",
    };
    restClient.Post<object>("/employees", newemp1);

    var emps = restClient.Get<List<Employee>>("/employees");

    var emp = emps.First();

    var empPatch = new Operations.EmployeePatch();
    empPatch.Add(new Operations.JsonPatchElement()
    {
        op = "replace",
        path = "/title",
        value = "Kahuna Laau Lapaau",
    });

    empPatch.Add(new Operations.JsonPatchElement()
    {
        op = "replace",
        path = "/cubicleno",
        value = "32",
    });

    restClient.Patch<object>(string.Format("/employees/{0}", emp.Id), empPatch);

    var empAfterPatch = restClient.Get<Employee>(string.Format("/employees/{0}", emp.Id));

    Assert.NotNull(empAfterPatch);
    // patched
    Assert.Equal("Kahuna Laau Lapaau", empAfterPatch.Title);
    Assert.Equal("32", empAfterPatch.CubicleNo.ToString());
    // unpatched
    Assert.Equal("test1@example.com", empAfterPatch.Email);
}

I am uploading this code to github a full working Visual Studio 2013 project, including xUnit.net tests.

I hope this has been useful to demonstrate the flexibility of using ServiceStack and C# to implement the HTTP PATCH method using JSON Patch (RFC 6902) over the wire.

Update: i refactored the code so that any object can have it’s properties “patched” from a JsonPatchRequest DTO by using an extension method populateFromJsonPatch().

public object Patch(EmployeePatch dto)
{
    // partial updates
    // get from persistent data store by id from routing path
    var emp = Repository.GetById(dto.Id);

    if (emp != null)
    {
        // update values which are specified to update only
        emp.populateFromJsonPatch(dto);

        // update
        Repository.Store(emp);

 

 

ServiceStack CSV serializer with custom filenames

One of ServiceStack’s benefits is having one service method endpoint output to all supported serializers. The exact same code will output formats for JSON, XML, CSV, and even HTML. If you are motivated, you are also free to add your own.

Now in the case of CSV output, the web browser handling the download will prompt the user to save the text/csv stream as a file. The ‘File Save’ dialog will fill in the name of the file, if it is included in the HTTP response this way:

todos-service-opening-dialog

Note the filename is “Todos.csv”, because the request operation name is “Todos”. (i’m using the example service code).

There could be many cases where you would like to have much more fine-grained control of the default filename. However, you don’t want to pollute the Response DTO, since that would ruin the generic “any format” nature of the framework. You’ll probably also want to be able to have different filename-creation logic per-service, since you’ll often have many services in one application.

In my attempt to get to the bottom of this,

  • I create a new blank ASP.NET project. The version i want is the 3.9.* version, since i’m not up on the v4 stuff.
  • Using this site, i can identify the correct version of the NuGet package, and install the correct ones: https://www.nuget.org/packages/ServiceStack.Host.AspNet/3.9.71
  • Then i install from the console.
    PM> Install-Package ServiceStack.Host.AspNet -Version 3.9.71
  • I see all my references are 3.9.71
  • My web.config has the ServiceSTack handlers installed, and my project has the App_Start\AppHost.cs

The demo project is the ToDo list. I’ll use it to test the CSV output. First, add a few items:

todos-ux

Then try to get the service ‘raw’:

http://localhost:49171/todos

You will see the generic ServiceStack output page:

todos-service-html

Next, click the ‘csv’ link on the top right, in order to get the service with a ‘text/csv’ format. You will get the prompt dialog, as shown at the top of this post, with the ‘Todos.csv’ filename.

if you inspect the HTTP traffic in Fiddler, the request is:

GET /todos?format=csv HTTP/1.1

the response of looks like this:

HTTP/1.1 200 OK
 Cache-Control: private
 Content-Type: text/csv
 Vary: Accept-Encoding
 Server: Microsoft-IIS/8.0
 Content-Disposition: attachment;filename=Todos.csv
 X-Powered-By: ServiceStack/3.971 Win32NT/.NET
 X-AspNet-Version: 4.0.30319
 X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNccmF1bGdcRG9jdW1lbnRzXGVuZWhhbmFcY29kZVxFbmVoYW5hLkNvZGVTYW1wbGVzXFNzQ3N2RmlsZW5hbWVcdG9kb3M=?=
 X-Powered-By: ASP.NET
 Date: Tue, 11 Mar 2014 01:43:52 GMT
 Content-Length: 88
Id,Content,Order,Done
 1,Get bread,1,False
 2,make lunch,2,False
 3,do launtry,3,False

The Content-Disposition: header defines the default filename of the save dialog box.

So how is this set? ServiceStack’s CSV serializer code sets is explicitly.

The best way i’ve discovered to do this is to plug in your own alternative CsvFormat plugin. If you view the source code, you’ll see where it sets the Content-Disposition: header in the HTTP Response.

    //Add a response filter to add a 'Content-Disposition' header so browsers treat it natively as a .csv file
    appHost.ResponseFilters.Add((req, res, dto) =>
    {
	    if (req.ResponseContentType == ContentType.Csv)
	    {
		    res.AddHeader(HttpHeaders.ContentDisposition,
			    string.Format("attachment;filename={0}.csv", req.OperationName));
	    }
    });

The docs for ServiceStack’s CSV Format are clear on it:

https://github.com/ServiceStackV3/ServiceStackV3/wiki/ServiceStack-CSV-Format

A ContentTypeFilter is registered for ‘text/csv’, and it is implemented by ServiceStack.Text.CsvSerializer.

Additionally, a ResponseFilter is added, which adds a Response header. Note the Content-Disposition: header is explicitly using the Request ‘OperationName’ as the filename. Normally this will be the Request DTO, which in this case is named ‘Todos’.

res.AddHeader(HttpHeaders.ContentDisposition, 
        string.Format("attachment;filename={0}.csv", req.OperationName));

So, what if we want to replace the default registration with different logic for setting the filename? We won’t need to change the registered serializer (still want the default CSV), but we should remove the ResponseFilter and add it in a slightly different way.

If you want to remove both, you can remove the Feature.Csv. However, in this case i just want to change the filter. I had trouble altering the response filter directly, so instead i created my own ‘CsvFilenameFormat’, which looks almost exactly like ‘CsvFormat’. The difference is that i try to get a custom filename from the service code, by looking in Request.Items Dictionary<string, object>.

The differing code in CsvFilenameFormat.Register():

    //Add a response filter to add a 'Content-Disposition' header so browsers treat it natively as a .csv file
    appHost.ResponseFilters.Add((req, res, dto) =>
    {
        if (req.ResponseContentType == ContentType.Csv)
        {
            string csvFilename = req.OperationName;

            // look for custom csv-filename set from Service code
            if( req.GetItemStringValue("csv-filename") != default(string) )
            {
                csvFilename= req.GetItemStringValue("csv-filename");
            }

            res.AddHeader(HttpHeaders.ContentDisposition, string.Format("attachment;filename={0}.csv", csvFilename));
        }
    });

So if the service code sets a custom value, it will be used by the text/csv response for the filename. Otherwise, use the default.

In the service:

        public object Get(Todos request)
        {
            // set custom filename logic here, to be read later in the response filter on text/csv response
            this.Request.SetItem("csv-filename", "customfilename");

So the mechanism is set up, all we need to do is properly prevent the default Csv ResponseFilter and use our own instead.

In AppHost Configure(), add a line to remove the Csv plugin, and one to install our replacement:

            // clear 
            this.Plugins.RemoveAll(x => x is CsvFormat);

            // install custom CSV
            Plugins.Add(new CsvFilenameFormat());

At this point, everything is in place, and we can re-run our web app:

todos-service-opening-dialog-customfilename

Project code here.

That’s the show. Thanks.

Customizing IAuthProvider for ServiceStack.net – Step by Step

Introduction

Recently, i started developing my first ServiceStack.net web service. As part of it, i found a need to add authentication to the service. Since my web service is connecting to a legacy application with its own custom user accounts, authentication, and authorization (roles), i decided to use the ServiceStack Auth model, and implement a custom IAuthProvider.

Oh yeah, the target audience for this post:

  • C# / .NET / Mono web developer who is getting started learning how to build a RESTful web api using ServiceStack.net framework
  • Wants to add the web API to an existing application with its own proprietary authentication/authorization logic

I tried to dive in and implement in my app, but i got something wrong with the routing to the /auth/{provider} , so i decided to take a step back and do the simplest thing possible, just so i understood the whole process.That’s what i’m going to do today.

I’m using Visual Studio 2012 Professional, but you could also use VS 2010, probably VS 2012 Express as well (or MonoDevelop, that’s another story i haven’t tried).

The simplest thing possible in my mind:

This is not an example of TDD-style development — more of a technology exploration.

OK, let’s get started.

Creating HelloWorld

I’m not going to repeat what’s already in the standard ServiceStack.net docs, but the summary is:

  • create an “ASP.NET Empty Web Application” (calling mine SSHelloWorldAuth)
  • pull in ServiceStack assemblies via NuGet (not my usual practice, but its easy). In fact, i’m using the “Starter ASP.NET Website Template – ServiceStack”. That will install all the assemblies and create references, and also update Global.asa
  • Create the Hello , HelloRequest, HelloResponse, and HelloService classes, just like the sample. Scratch that – it is already defined in the template at App_Start/WebServiceExamples.cs
  • Run the app locally. You will see the “ToDo” app loaded and working in the default.htm. Also, you can test the Hello function at http://localhost:65227/hello (your port number may vary)

 

Adding a built-in authentication provider

OK that was the easy part. Now we’re going to add the [Authenticate] attribute to the HelloService class.

[Authenticate]
public class HelloService : Service
{  ...

This will prevent the service from executing unless the session is authenticated already. In this case, it will fail, since nothing is set up.

Enabling Authentication

Now looking in App_Start/AppHost.cs , i found an interesting section:

		/* Uncomment to enable ServiceStack Authentication and CustomUserSession
		private void ConfigureAuth(Funq.Container container)
		{
			var appSettings = new AppSettings();

			//Default route: /auth/{provider}
			Plugins.Add(new AuthFeature(this, () => new CustomUserSession(),
				new IAuthProvider[] {
					new CredentialsAuthProvider(appSettings), 
					new FacebookAuthProvider(appSettings), 
					new TwitterAuthProvider(appSettings), 
					new BasicAuthProvider(appSettings), 
				})); 

			//Default route: /register
			Plugins.Add(new RegistrationFeature()); 

			//Requires ConnectionString configured in Web.Config
			var connectionString = ConfigurationManager.ConnectionStrings["AppDb"].ConnectionString;
			container.Register<IDbConnectionFactory>(c =>
				new OrmLiteConnectionFactory(connectionString, SqlServerDialect.Provider));

			container.Register<IUserAuthRepository>(c =>
				new OrmLiteAuthRepository(c.Resolve<IDbConnectionFactory>()));

			var authRepo = (OrmLiteAuthRepository)container.Resolve<IUserAuthRepository>();
			authRepo.CreateMissingTables();
		}
		*/

Let’s use it. But i want to just enable CredentialsAuthProvider, since that is a forms-based username/password authentication, (the closest to what i want to do customized).

A few notes on the code block above:

The “Plugins.Add(new AuthFeature(() ” stuff was documented.

“Plugins.Add(new RegistrationFeature());” was new to me, but now i see it is to add the /register route and behavior
For this test, i will go along with using the OrmLite for the authentication tables. In order to do that,

  • i’m using a new connection string “SSHelloWorldAuth”,
  • adding it to Web.config: <connectionStrings><add name=”SSHelloWorldAuth” connectionString=”Data Source=.\SQLEXPRESS;Initial Catalog=SSHelloWorldAuth;Integrated Security=SSPI;” providerName=”System.Data.SqlClient” /></connectionStrings>
  • creating a new SQLEXPRESS database locally, called: SSHelloWorldAuth

Finally, we’ll have to add/enable the line to ConfigureAuth(container) , which will initialize the authentication system.

Now we’ll try running the app again: F5 and go to http://localhost:65227/hello in the browser again. I get a new problem:

In a way, it’s good, because the [Authenticate] attribute on the HelloService class worked – the resource was found, but sent a redirect to /login . However, no handler is set up for /login.

Separately, i checked if the OrmLite db got initialized with authRepo.CreateMissingTables(); , and it seems it did (2 tables created).

Understanding /login , /auth/{provider}

This is where i got hung up on my initial try to get it working, so i’m especially determined to get this working.

The only example of a /login implementation i found in the ServiceStack source code tests. It seems like /login would be for a user to enter in a form. It seems if you are a script (javascript or web api client), you would authenticate at the /auth/{provider} URI.

That’s when i thought – is the /auth/* service set up properly? Let’s try going to http://localhost:65227/auth/credentials

So the good news is that is is set up. Why don’t we try to authenticate against /auth/credentials ?

Well, first i should create a valid username/password combination. I can’t just insert into the db, since the password must be one-way hashed. So i’m going to use the provider itself to do that.

I copied a CreateUser() function in the ServiceStack unit tests, and will run in my app’s startup. I modified slightly to pass in the OrmLiteAuthRepository, and call it right after initializing the authRepo.

CreateUser(authRepo, 1, "testuser", null, "Test2#4%");

Run the app with F5 again, and then check the database: select * from userauth — we now have one row with username and hashed password. Suitable for testing. (don’t forget to disable CreateUser() now).

Authenticating with GET

I would never do this on my “real” application. At minimum, i would only expose a POST method. But instead of writing some javascript, i’m going to try the web browser to submit credentials and try to authenticate.

First, i’m going to try and use a wrong password:

http://localhost:65227/auth/credentials?UserName=testuser&Password=wrong

… i get the same “Invalid UserName or Password” error, which is good.

Now i’ll try the correct username/password (url-encoding left as an exercise for the reader):

http://localhost:65227/auth/credentials?UserName=testuser&Password=Test2%234%25

Success! This means my user id has a validated ServiceStack session on the server, and is associated with my web browser’s ss-id cookie.

I can now go to the /hello service on the same browser session, and it should work:

Awesome. So we’ve figured out the /auth/credentials before the /hello service. Just for kicks, i stopped running the app in Visual Studio and terminated my local IIS Express web server instance, in order to try a new session. When i ran the project again and went to /hello , it failed as expected (which we want). Only by authenticating first, do we access the resource.

IAuthProvider vs IUserAuthRepository

Note that i started this saying i wanted to implement my own IAuthProvider. However, ServiceStack also separately abstracts the IUserAuthRepository, which seems to be independently pluggable. Think of it this way:

  • IAuthProvider is the authentication service code backing the HTTP REST API for authentication
  • IUserAuthRepository is the provider’s .NET interface for accessing the underlying user/role data store (all operations)

Since my initial goal was to use username/password login with my own custom/legacy authentication rules, it seems more appropriate to use subclass CredentialsAuthProvider (creating my own AcmeCredentialsAuthProvider).

I do not expect to have to create my own IUserAuthRepository at this time– but it would be useful if i had to expose my custom datastore to be used by any IAuthProvider. If you are only supporting one provider, you can put the custom code into the provider’s TryAuthenticate() and OnAuthenticated() methods. With a legacy system, you probably already have tools to manage user accounts and roles, so you’re not likely to need to re-implement all the IUserAuthRepository methods. However, if you need to implement Roles, a custom implementation of IUserAuthRepository may be in order (to be revisited).

This is going to be almost directly from the Authentication and Authorization wiki docs.

  • Create a new class, AcmeCredentialsAuthProvider.cs
  • subclass CredentialsAuthProvider
  • override TryAuthenticate(), adding in your own custom code to authenticate username/password
  • override OnAuthenticated(), adding any additional data for the user to the session for use by the application
    public class AcmeCredentialsAuthProvider : CredentialsAuthProvider
    {
        public override bool TryAuthenticate(IServiceBase authService, string userName, string password)
        {
            //Add here your custom auth logic (database calls etc)
            //Return true if credentials are valid, otherwise false
            if (userName == "testuser" && password == "Test2#4%")
            {
                return true;
            }
            else
            {
                return false;
            }
        }

        public override void OnAuthenticated(IServiceBase authService, IAuthSession session, IOAuthTokens tokens, Dictionary<string, string> authInfo)
        {
            //Fill the IAuthSession with data which you want to retrieve in the app eg:
            session.FirstName = "some_firstname_from_db";
            //...

            //Important: You need to save the session!
            authService.SaveSession(session, SessionExpiry);
        }
    }

As you can see, i did it in a trivially stupid way, but any custom logic of your own will do.

Finally, we change AppHost.cs ConfigureAuth() to load our provider instead of the default.

			Plugins.Add(new AuthFeature(() => new CustomUserSession(),
				new IAuthProvider[] {
					new AcmeCredentialsAuthProvider(appSettings), 
				}));

Run the app again, you should get the same results as before passing the correct or invalid username/password. Except in this case, you can set a breakpoint and verify your AcmeCredentialsAuthProvider code is running.

So at the end of this i’m happy:

  • I established how to create a ServiceStack service with a working custom username/password authentication
  • I learned some things from the ServiceStack Nuget template which was in addition to the docs
  • I understand better where it is sufficient to only override CredentialsAuthProvider for IAuthProvider , and where it may be necessary to implement a custom IUserAuthRepository (probably to implement custom Roles and/or Permissions)

Thanks for your interest. If you are interested in the code/project file created with this post, i’ve pushed it to GitHub.