Plugin architecture (I): Preliminary notes

Tags: , , , ,

This is the first article of a three-part series, read the second and third article: Plug-in architecture II: JavaScript Event-Driven plug-in framework and C++ Event-Driven Plug-in Framework. (Plug-in architecture III).

Our application needed the integration of code made by third party developers in the client and in the server side. Our application is written in HTML, CSS and javascript in the client side, the core in the server side is written in C++. But I will also post code for servers using PHP.

In this set of articles I explain how we designed and integrated our plugin framework in a web application project. This first article is a resume of existing plugin architectures and our plugin design. In the next articles I will document the implementation of the described plugin architecture and the development of some plugins to equip our web application with new functionalities.

What do we want

  • Each of the application users will use different sets of plugins.
  • Plugins will be installed, updated and uninstalled without having to restart the server.
  • We need communication between the client side of a plugin and its server side. And communication between plugins in the same side.
  • Allow users to modify or override existing plugin functions.
  • Plugins will be executed asynchronously.
  • No need of inheritance hierarchy: plugins can be completely independent, no obligation to use core application's functions.
  • Plugins will be shared, and a user may not know the plugin code or if the provider is secure, so we will need to have some kind of security protocols.

Examples of plugin architectures

emacs

  • Website: https://www.gnu.org/software/emacs/
  • Core written in C.
  • Emacs is an interpreter for Lisp. So it is very easy to extend emacs. Extensions are written in Emacs Lisp and extend the existing code.
  • A command in emacs is a Lisp function and can be invoked with M-x command-name. There are different commands, for managing windows, text management etc. that can also be used in our Lisp extensions.
  • There are events for "catching" the keyboard and mouse activity.
  • Hooks: We can see hooks as events. "A hook is an ordinary Lisp variable whose value is a list of functions that get executed under specific conditions." Example: make buffer read-only when visiting symbolic link file. The action of visiting a file should trigger a function I wrote. This is where hooks come in. find-file-hooks is the hook every time a new file is visited (There are many hooks). Read more about Elisp hooks.
  • Modes: Define basic behavior of the application. For example "Text mode is a major mode for editing human languages", HTML mode is mode for editing HTML documents. A mode can be derived from another mode.
  • Macros: Macros enables you to define new control constructs and other language features. A macro is defined much like a function, but instead of telling how to compute a value, it tells how to compute another Lisp expression which will in return compute the value. We call this expression the expansion of the macro. Read more about Elisp macros.

eclipse

  • Website: http://www.eclipse.org/
  • Based on Equinox (OSGi framework).
  • Plugins are updated without restart. And eclipse mantains the plugins automatically.
  • Plugins are in form of package, containing plugin descriptors, plugin resources, implementation code...
  • A plugin has different XML files for describing itself and its requirements. Example: "The plug-in manifest file, plugin.xml, describes how the plug-in extends the platform, what extensions it publishes itself, and how it implements its functionality. The manifest file is written in XML and is parsed by the platform when the plug-in is loaded into the platform. All the information needed to display the plug-in in the UI, such as icons, menu items, and so on, is contained in the manifest file. The implementation code, found in a separate Java JAR file, is loaded when, and only when, the plug-in has to be run. This concept is referred to as lazy loading." Read more here.
  • Plugins implementation code is written in Java.
  • A plugin can depend on other plugins.
  • A plugin can extend other plugin functionalities.
  • A plugin declare it is extesive "through" an extension point: "When a plug-in wants to allow other plug-ins to extend or customize portions of its functionality, it will declare an extension point. The extension point declares a contract, typically a combination of XML markup and Java interfaces, that extensions must conform to. Plug-ins that want to connect to that extension point must implement that contract in their extension. The key attribute is that the plug-in being extended knows nothing about the plug-in that is connecting to it beyond the scope of that extension point contract. This allows plug-ins built by different individuals or companies to interact seamlessly, even without their knowing much about one another." Read more about what are extensions and extension points.
  • You can inject Services to the plugin. Services are software components based on an interface class which provide functionality. Example: ECommandService gives access to existing host application commands and allows you to create and change commands.
  • A plugin can use listeners for knowing the state of other plugins. Example: "The host plug-in defines a menu item, which, when selected, causes the state of the subject to be updated. In turn, the state change in the subject causes listener notifications to be broadcast to each listener configured into the system." Read more about listeners

Chrome

    Extensions:
  • Extends functionality of the chrome browser.
  • Extensions are coded in HTML, CSS and Javascript.
  • A button (Browser action or Page action) in the top of the browser gives access plugin.
  • Has a manifest for describing itself. Read Manifest file format.
  • Internationalization of variables (chrome.i18n), so they are translated in different languages.
  • Communication through some kind of events and listeners. There is intra-extension and inter-extension communication. Read more about message passing in chrome extensions.
  • Can do cross server requests.
  • Oauth authentication supported in extensions.
  • Extensions vs Plugins:
  • Plugins display not nativelly supported content. A plugin example is a plugin for watching videos or flash contents.

Mozilla

    Add-ons:
  • Extensions: In javascript, based on nodejs.
  • Web Extensions: similar to chrome (they are working on it)
  • Plugins:
  • Same as chrome, used for displaying non native supported content.

Useful resources

Our plugin design

Properties of our plugin framework

Here is a table grouping the properties of our plugin framework.

Property Description
Implementation code language - Client side: javascript.
- Server side: Compiled application?, embeded PHP, Lisp, CSL, or Javascript?. ???
Plugin file structure - Manifest (Plugin declaration): Plugin description and requirements.
- Configuration file.
- Resources: Images, JSON files, CSS, HTML...
- Script or executable.
- Temporary files.
- Documentation
C++ and Javascript Plugin package structure
Life of a plugin - Install.
- Update / mantain plugin.
- Uninstall.
All of them without having to restart the application.
Lifecycle during application runtime - Load plugin (lazy loading | eager loading.), even some kind of "require" to load plugin before having to use it, to speed the application.
- Run plugin functions asynchronously.
- Remove plugin.
Plugin lifecycle will be managed by a Plugin Handler.
Trigger - By events. We will use our own event management, we are not using javascript native events.
- By plugin Id.
- pluginA can fire an event that will execute a certain function in pluginB, this function will recieve pluginA data and do something (this data is somehow standardized).
By default when triggering a plugin, run(data) function is executed if no plugin function specified. If no event specified plugin will be triggered when loaded, anyway when plugins will be loaded we will fire "load-plugins-after" event.
Communication - Communication client-server (HTTP).
- Communication with other plugins: parameters, pipes, network based communication, events.
Communication using JSON with some kind of standard structure.
Acces other plugin state - Access pluginB from pluginA with pluginB Id.
Data A plugin manipulates data, returns data.
Override Override other plugins functions.
Security - scripts are more secure, code can easily be reviewed.
- Is it secure to have compiled plugins? => Use a script interpreter?
Documentation - Technical documentation.
- User documentation.

Our plugins are not defined exclusively as client or server side, they can be both.
They will be installed, updated and uninstalled without the need of restarting the application
Each plugin will be a package containing description files (manifest), configuration files, executables or scripts, resources, temporary files and documentation.
There will be a Plugin Handler in both client and server side to manage the lifecycle of plugins during the application runtime.
The Plugin Handler manage the plugin events. The Plugin Handler will call the plugins functions that have to be called when a certain event is fired.
There will be a client-server plugin communication, so we will need a controller that will handle the client requests and establish communication with the plugins in the server side. This Plugin Controller will also deliver the client plugins when asked.
There will also be a communication between plugins in the same side.

Plugin framework components and flows

Here is a diagram showing the plugin framework components and its flows. How plugin framework works

Problems found

All my issues have to do with the server side. As I said it is written in C++. We develop using C++ REST SDK and...

  • Each user of our application will use different plugins and plugin configuration, and C++ REST SDK has no session management.
  • There are no examples of authentication server in C++ REST SDK.
  • It is difficult to develop compiled plugins in the server side, if we use compiled code there can be security issues.

In the "plug-in architechture II" article I talk about the client side plugin framework documentation, examples and code.

C++ REST SDK (Casablanca) Multipart form data HTTP request parser

Tags: , ,

Objective

We want to parse an http request for working with multipart/form data in a C++ REST SDK (Casablanca) project.

The problem

There have been questions about how to upload an image file using C++ REST SDK (multipart form POST example, http_listener and multipart/form-data, Multipart request handling in the server side using http listener)

But for the moment there is no integrated multipart parsing support. That is why we developed our own simple parser. I'm going to explain how it works so you can download and use it. Feel free to modify it, comment and ask if you have problems.

Download Parser

parser.h: granada::http::parser header
parser.cc: granada::http::parser cc

Let's cook

What we are going to do is send the data of a form containing an image file via AJAX to our REST server via an HTTP POST request. Then in the server side we are going to parse the http_request, save our image file in our hard drive and finally respond to the client.

Client side

We use an HTML form with a file field and another text field as an example:

<form enctype="multipart/form-data">
	<input type="text" name="test-text-field" value="test value" />
	<input type="file" name="file"/>
  <button type="button" id="upload-button">Upload</button>
</form>

We send the form, it can be done via AJAX. To simplify, here is the code using jQuery:

$('#upload-button').click(function(){
	var formData = new FormData($('form')[0]);
	$.ajax({
		url: "/test",
		type: "POST",
		success: function(data){
			console.log('data => ', data);
		},
		data: formData,
		cache: false,
		contentType: false,
	  processData: false
  });
});

Server side

Here is how we parse the data in a post handler function. The function receives an http_request containing our file field and our text field we just sent, parses it into an unordered_map, then extracts the file from our parsed data, saves the file in the hard drive and responds with a json to the client. For that purpose we will use the http_request parser (parser.h and parser.cc):

void TestController::handle_post(http_request request){

      // parse multipart/form data content of http request
      std::unordered_map<std::string, std::unordered_map<std::string, std::vector<unsigned char>>> multipart_form_data = granada::http::parser::ParseMultipartFormData(request);

      // extract file field from parsed data
      std::unordered_map<std::string, std::vector<unsigned char>> file_properties = multipart_form_data["file"];
      std::vector<unsigned char> file_name_and_extension = file_properties["filename"];
      std::string file_name_and_extension_str(file_name_and_extension.begin(),file_name_and_extension.end());

      // build file path
      std::string file_path = "/path/to/images/directory/" + file_name_and_extension_str;

      // file where image is going to be.
      std::ofstream FILE(file_path, std::ios::out | std::ofstream::binary);

      // extract file data and save it into file.
      std::vector<unsigned char> file_vector = file_properties["value"];
      std::copy(file_vector.begin(), file_vector.end(), std::ostreambuf_iterator(FILE));

      // response with the path of the saved file.
      request.reply(status_codes::OK, U("{\"success\":true,\"message\":\"File uploaded.\",\"data\":{\"src\":\"" + file_path + "\"}}"), "text/json; charset=utf-8");
}
That's it!

How the parser works

Example of an HTTP request:

Accept: */*
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 6500
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarymBItcSVphgAjmbJC
Host: localhost
Origin: http://localhost
Pragma: no-cache
Referer: http://localhost/?file=test-image.png
User-Agent: Mozilla/5.0 (X11; Linux x86_64)(...)
X-Requested-With: XMLHttpRequest
------WebKitFormBoundarymBItcSVphgAjmbJC
Content-Disposition: form-data; name="test-text-field"

Test value
------WebKitFormBoundarymBItcSVphgAjmbJC
Content-Disposition: form-data; name="file"; filename="test-image.png"
Content-Type: image/png

\89PNG
#
\00\00\00
IHDR\00\00\00d\00\00\00^##\00\00\00\D5#\B1\9F\00\00\00#
(…)
------WebKitFormBoundarymBItcSVphgAjmbJC--
What parser does:
1- Get boundary that separates each block of fields from headers so we can fragment the request body:
WebKitFormBoundarymBItcSVphgAjmbJC
2- Remove headers and work only with the body of the HTTP request:
------WebKitFormBoundarymBItcSVphgAjmbJC
Content-Disposition: form-data; name="test-text-field"

Test value
------WebKitFormBoundarymBItcSVphgAjmbJC
Content-Disposition: form-data; name="file"; filename="test-image.png"
Content-Type: image/png

\89PNG
#
\00\00\00
IHDR\00\00\00d\00\00\00^##\00\00\00\D5#\B1\9F\00\00\00#
(…)
------WebKitFormBoundarymBItcSVphgAjmbJC--
3- Parse each block of fields: extract the properties of the field and its value and insert them into an unordered_map of string and vector of unsigned char.Then Insert the unordered_map of properties into the multipart_form_data unordered_map that contains all the fields:
parsed multipart form (unordered_map)
	|_ test-text-field (unordered_map)
	|	|_ value (vector)
	|
	|_ file (unordered_map)
		|_ value (vector)
		|_ filename (vector)
		|_ Content-Type (vector)

Helpful links

Classes References

http_request: web::http::http_request Class Reference
http_headers: web::http::http_headers Class Reference
http_response: web::http::http_response Class Reference

More

Hypertext Transfer Protocol – HTTP/1.1 W3
using HTTP POST to upload Image
How to set http_requset body in an HTTP POST with content type "multipart/form-data"