Development on dotnet.work

How to track a squirrel

dotnet@oles.cloud (Ole Albers) — Sun, 17 Dec 2023 12:00:00 +0100

[Disclaimer: This is not meant as a tutorial, more as some chitchat. You can find the source on github nonetheless, but I assume this is a VERY specific use case noone else uses]

Let’s start by telling you that this whole project ist utter nonsense. Noone really needs this feature. I decided to create this nonetheless.

The Goal

I own a little feeding box for squirrels of the neighbourhood. And it works as designed: Squirrels get here, steal their food and run away.

I can see them through my glass door and it just looks amazing, especially in the typical “Hamburger Wetter” (Hamburg weather, means: raining horizontally). Sometimes I take a picture or even a video, but through the door that just does not look very great.

So I decided to add a surveillance cam to the mix, and while I am on it: Automatically publish whenever a squirrel appears

Hardware

I use a dlink - outdoor webcam with WiFi. Not really worth looking up the model number here, just the cheapest WiFi-Cam I could find which is compatible with Synology Surveillance Station (but nearly all are)

In addition the Synology NAS with installed surveillance station. I own a DS920+, but any should do

Privacy

I do not want to track any persons. Not me, not others. This is not meant to be surveillance, it is meant to film squirrels. TBH I did not ask the squirrels how they think about privacy and GDPR, but I assume they are fine with it. After all they accepted all Cookies until now, and not only the required ones.

To make sure only squirrels (or other animals) get caught I wanted to do a few conditions:

Only track movement directly in front of the feeder.
- So I can make sure not Humans or Traffic in the background trigger the motion sensor
Put the Camera in “Squirrel”-height.
- So even if someone (me or my guests) manages to trigger the alarm no faces can be seen
Have a simple to use approval workflow
- Just in case. If someone ends up lying there drunk I do not want to publish a video of that person anywhere

Only Track movement directly in front of the feeder

That is one of the reasons why I use the Surveillance Station of Synology. The Camera itself has movement detection and (according to the documentation) even a fancy AI “this is a person” algortihm. I do not want the last for obvious reasons and especially can not decide which area of the image should be used for detection. In addition this seems to require a cloud feature. I do not want my camera feed in the cloud from anyone. This should never leave my house without my consent. The surveillance station can select very small pieces of the image to be selected as alert regions. I put it directly in front of the feeder where no background movement can be detected. This way I can also adjust the sensibility of the detection to quite a high value because no noise becaues of branches moving in the wind etc. is to be expected.

Put the camera in squirrel-height

That was easy as there are some flowerboxes directly before the feeder where I mounted the cam

Have a simple approval workflow

Because I mostly am at Mastodon at time of this writing (these sentences sometimes age very bad. A few months ago that would have been the artist formerly known as Twitter, which itself was just a replacement for Google+ which, well, you know the sad story). Let’s just hope that was the last Social Media switch for me.

One of the advantages of Mastodon is that it has a very nice, simple API that allows sending toots, getting Notifications and so on. So my idea for the workflow is to notify me (and only me) that a new movement has been detected. Attach an image of that movement and if I approve this (e.g. by adding a “fav” to that toot) the bot knows that this detection was a valid Squirrel video and posts it. So AI without the “A”. (some might argue even without the “I”…)

Synology

Adding the Webcam is quite simple. It nearly is found automatically in Surveillance station. Finding out how to connect was a bit harder. Username is always “admin” and the required password is a PIN. Printed onto the Power Plug, not the device… After that the App of the camera does not like the Camera anymore, which is bonus for me as I do not want images to be uploaded to dlink, anyways.

The detection area can be set in the Camera options. Fiddling with the thresholds takes a while, especially as I do not have a trained squirrel to assist me on that. I then changed the time plan to always motion detection (so disabling the constant recording) and updated the minimum record time to five minutes. A good comprimise between getting everything and not creating videos that are too big for mastodon/youtube/wherever I wanted to upload this, which I had not decided at this point

Synology has multiple ways to notify about movements: Email, SMS, Push notifications and Webhooks. Obviously the last one was the way to go as it does not spam me in any way and I have to program stuff anyways.

Writing a Webhook

A webhook is just an HTTP-call essentially. Synology allows POST and PUT in Surveillance Station. So I created a simple Web Api in Visual Studio and was ready to debug the incoming data.

Firewalls, Firewalls everywhere!

Synology allows you to send a test-notification which just resulted in an error. WHAT exactly went wrong is nowhere to be seen. Not in surveillance station and not in the logs from the synology. So I had to fly blind at this point.

I knew it worked when using my local machine, either via localhost or the ipadress from my development machine.

So the next step was to connect to my synology via SSH and try to access my API throuh a simple webcall via GET: wget https://192.168.178.23:6789/ping

This simple call already did not work. So must have been a firewall issue. I checked everything in my Synology configuration FOR A VERY LONG TIME and did find nothing. Finally I put my attention to Windows, disabled the Defender Firewall and it went through. Because I do not want to have the firewall open I added an exception through the “allow an app through windows firewall” and finally it did work with the firewall still enabled.

Receiving data from the webhook

The Webhook arrived as expected, has some informations in it and can even provide an url to a preview image. That image URL is a generated “public” (within my LAN) Address I am able to receive and download:

app.MapPost("/squirrel", async context =>
{
    var requestBody = await context.Request.ReadFromJsonAsync<SynologyBody>();
    if (requestBody != null)
    {
        if (!string.IsNullOrWhiteSpace(requestBody.Capture))
        {
            var previewImage = await downloader.DownloadFile(requestBody.Capture);                                    
        }
        else
        {
            Console.WriteLine("received event without image. Will ignore it");
        }
    }
    await context.Response.WriteAsync("event received");
});

public class SynologyBody {
  public string Subject { get; set; } 
  public string Desc { get; set; } 
  public string Capture { get; set; } 
}

(Synology does not really care about the response but I found it useful when testing locally through Postman)

At this point I do not have a video yet which is to be expected, as “normal” users would expect to receive the warning immediately and not five minutes after someone stole their nuts.

Send private notification to Mastodon

After an event have been received the Image is downloaded and a message is sent to Mastodon using MastoNet:

private MastodonClient Login()
{
    return new MastodonClient(_instance, _accessToken);
}

public async Task<Status?> SendPreview(Stream previewFile, DateTime notificationDate)
{
    var client = Login();
    var text = $"{_approver} ALARM! Possible Squirrel detection! ALARM! #preview DATE:\n{notificationDate}\n";
    var media=await client.UploadMedia(previewFile, "squirrel.png", "possibly a squirrel");
    var status=await client.PublishStatus(text, Visibility.Direct, mediaIds: new[] {media.Id }); 
    if (string.IsNullOrWhiteSpace(status.Id))
    {
        Console.WriteLine("Could not send preview");
        return null;
    }
    Console.WriteLine($"Published to Mastodon with id {status.Id}");
    return status;
}

This creates the following toot:

Waiting for approval

This toot has only been sent to me privately (mastodon has no encryption so always keep that in mind about the “privacy” - part). I can then decide that this is a valid Squirrel detection by setting the “fav” star of that toot.

Keeping the state

To be able to understand which previewtoot belongs to what webhook call I need to store the state. I uses LitDB for this. LiteDB is a very simple document database which needs no installation at all. It completely only lives within the NuGet-Package and the data is stored in a Bson-Format (Json, but compressed). It is a bit like simply serializing my contents and store it into a Json file. Just performing much better.

That is why the complete “database”-class is as simple as this:

 public class DataBase
    {
        private const string _databaseName = @"Filename=squirrel.db;connection=shared";

        public T GetContents<T>(string id)
        {
            using var db = new LiteDatabase(_databaseName);
            return db.GetCollection<T>().FindById(new BsonValue(id));
        }

        public void RemoveContents<T>(string id)
        {
            using var db = new LiteDatabase(_databaseName);
            db.GetCollection<T>().Delete(new BsonValue(id));
        }

        public void UpsertContents<T>(T contents)
        {
            using var db = new LiteDatabase(_databaseName);
            db.GetCollection<T>().Upsert(contents);
        }

        public SquirrelData? FindByFilename(string filename)
        {
            using var db = new LiteDatabase(_databaseName);
            return db.GetCollection<SquirrelData>().FindOne(q => q.ExpectedMovieFile == filename);
        }
    }

Now the data inside the Database contains the following stuff:

public class SquirrelData
    {
        [BsonId]
        public string Id { get; set; }  // First MastodonID
        public string? PublicId { get; set; }

        public DateTime EventReceived { get; set; }
        public DateTime SentForApproval { get; set; }
        public DateTime? SentPublic { get; set; }
        public DateTime? Faved { get; set; }
        public string? ExpectedMovieFile { get; set; }
    }

The Id simply is the Id from the message just posted to mastodon. This object mostly contains Mastodon-Information and Information about the WebHook event.

As mentioned above the WebHook has no information about the Video that is created (or most likely: Is still being created when the WebHook is sent). So we simply have to find it. The rules are quite simple: Within the directory with surveillance videos it is simply the newest file we can find.

One problem remains: The preview image was within the WebHook and did not need authentication do be downloaded. The surveillance directory does need authentication. So I created a user for that specific purpouse allowed it to access the folder and needed to do some SMB-Stuff for authentication.

I used the SMBLibrary for that. It isn’t as straight forward as navigating through a filesystem, but it does work fine. First I have to connect using my credentials:

 public void Connect()
{
    var smbClient = new SMB1Client(); 
    bool isConnected = smbClient.Connect(IPAddress.Parse(_server), SMBTransportType.DirectTCPTransport);
    if (isConnected)
    {
        NTStatus status = smbClient.Login(string.Empty, _username, _password);
        if (status == NTStatus.STATUS_SUCCESS)
        {
            _smbClient=smbClient;
            return;
        }
    }
    throw new Exception("Connecting failed. Check your credentials");
}

After successfully connecting I can access the share and search for the most current mp4-file recursively:

     
public string FindMostCurrentVideo()
{
    var store = _smbClient.TreeConnect("surveillance", out var status);
    if (status != NTStatus.STATUS_SUCCESS) throw new Exception("Connecting to Share failed");
    var mostCurrentFile=GetMostCurrentFileRec((SMB1FileStore)store, _startPath);
    return Path.Combine(mostCurrentFile.Path, mostCurrentFile.Filename);
}

private MostCurrentFile GetMostCurrentFileRec(SMB1FileStore store, string folder)
{
    folder += "\\";
    var mostCurrentFile = new MostCurrentFile { Created = DateTime.MinValue };

    var status=store.QueryDirectory(out var subdirectoryList, folder, SMBLibrary.SMB1.FindInformationLevel.SMB_FIND_FILE_DIRECTORY_INFO);
    if (status== NTStatus.STATUS_SUCCESS)
    {
        object directoryHandle;
        FileStatus fileStatus;
        status = store.CreateFile(out directoryHandle, out fileStatus, folder, AccessMask.GENERIC_READ, SMBLibrary.FileAttributes.Directory | SMBLibrary.FileAttributes.Normal, ShareAccess.Read | ShareAccess.Write, CreateDisposition.FILE_OPEN, CreateOptions.FILE_DIRECTORY_FILE, null); ;
        if (status == NTStatus.STATUS_SUCCESS)
        {
            List<FindInformation> fileList2;
            status = store.QueryDirectory(out fileList2, folder+ "*", FindInformationLevel.SMB_FIND_FILE_DIRECTORY_INFO);
            var dirInfo = fileList2.Where(q => q.GetType() == typeof(FindFileDirectoryInfo)).Cast<FindFileDirectoryInfo>();
            var subdirectories=dirInfo.Where(q=>q.ExtFileAttributes.HasFlag(ExtendedFileAttributes.Directory));
            var files=dirInfo.Where(q => !q.ExtFileAttributes.HasFlag(ExtendedFileAttributes.Directory));
            // Look for subdirectories and drill down to them:
            foreach (var directory in subdirectories)
            {
                if (directory.FileName.Contains(".")) continue;
                var mostCurrentSubFile = GetMostCurrentFileRec(store, Path.Combine(folder, directory.FileName));
                if (mostCurrentSubFile.Created > mostCurrentFile.Created) mostCurrentFile = mostCurrentSubFile;
            }

            // Look for .mp4 - Files in current directory and check if they are newer than the most current file known:
            foreach (var file in files.Where(q=>q.FileName.EndsWith(".mp4")))
            {
                if (file.CreationTime > mostCurrentFile.Created)
                {
                    mostCurrentFile = new MostCurrentFile
                    {
                        Created = file.CreationTime.Value,
                        Filename = file.FileName,
                        Path = folder
                    };
                }
            }
            status = store.CloseFile(directoryHandle);
        }
    }
    return mostCurrentFile;
}

At this point I do not need the filecontents itself, but only the path so I can use it lateron.

After that we have the whole WebHook complete:

Reveive the Contents (including preview image) from the webhook
Determine which Video File matches
CHeck if we already had this file (if within a video multiple webhooks are fired) and avoid duplicates
Send a notification to Mastodon
Store all known information into our database.

So our Webhook from the beginning now looks like this:

app.MapPost("/squirrel", async context =>
{
  var received = DateTime.Now;
  Console.WriteLine("Received squirrel Event");
  var requestBody = await context.Request.ReadFromJsonAsync<SynologyBody>();
  if (requestBody != null)
  {
    if (!string.IsNullOrWhiteSpace(requestBody.Capture))
    {
      var previewImage = await downloader.DownloadFile(requestBody.Capture);
      downloader.Connect();
      string videoFilename = downloader.FindMostCurrentVideo() ?? throw new Exception("No Video Found");
      var existingItem = database.FindByFilename(videoFilename);
      if (existingItem != null)
      {
        Console.WriteLine($"'{videoFilename}' already processed with MastodonId {existingItem.Id}");
        await context.Response.WriteAsync("duplicate event");
        return;
      }
      var mastodonStatus = await mastodon.SendPreview(previewImage, received);
      if (mastodonStatus != null)
      {
        database.UpsertContents(new SquirrelData { EventReceived = received, ExpectedMovieFile = videoFilename, Id = mastodonStatus.Id, SentForApproval = mastodonStatus.CreatedAt });
      }
    }
  }
  await context.Response.WriteAsync("event received");
});

Checking for Notifications

As mentioned the “approval workflow” (a big word for this simple mechanism, I know) is to wait for a fav. If the fav is received it means I decided this is a valid Squirrel detection and the video can be published.

Also do not wait longer than 7 days. If I did not approve in that time, it probably is not a good video.

Again using MastoNet I can receive the Notifications for that user:

public async Task<List<Notification>> GetFavs()
{
  var favourites= new List<Notification>();
  var client=Login();
  var notifications=await client.GetNotifications();
  foreach (var notification in notifications)
  {
    switch (notification.Type)
    {
      case "mention":
        await client.PublishStatus($"Sorry. I am just a bot and will not respond to this. Ask {_approver} if you want to know anything about what I do", replyStatusId: notification.Id);
        await client.DismissNotification(notification.Id);
        break;
      case "favourite":
        favourites.Add(notification);
        break;
      default:
        Console.WriteLine($"discarding notification of type {notification.Type}");
        await client.DismissNotification(notification.Id);
        break;
    }
  }
  return favourites;
}

The nofication.Status.Id now is the id we already stored in the Database and can use to find all known WebHook-Data of that event and send a new Toot. This time just with the Video file instead of the preview image.

Why no Mastodon Webhooks?

The Mastodon API also offers Webhooks. So instead of periodially polling for new notifications this could be used to receive the favs. The main reason why I did not use this (apart from the fact that polling is simply fast enough) is the fact that my application is not accessible from the outside. To allow this I would need to make my NAS (where the application is hosted) accessible from the Internet. And I do not want that. I could add Firewall rules to only allow specific ports and so on, but no access is even a bit better than a limited access. Even if noone would get through, just bots trying the flood my machine with requests is a headache I do not need.

Uploading the Video

After trying to uplaad the Video directly to Mastodon the next problem came up: The file was too big. While it was way inside the limits the upload (to the mastodon.social instance) just responded with an error “something went wrong”. I suspect the Library to be the main issue as the Upload is not done async and so a timeout might be the cause.

That is why I decided to upload the Video to YouTube instead. YouTube provides APIs to do that. The main issue is that this also requires an OAuth authentication. Luckily the documentation for this are quite straight-forward so a simple Test worked fine:

public async Task UploadVideo(Stream fileStream)
{
  var youtubeService = new YouTubeService(new BaseClientService.Initializer()
  {
      ApiKey = _apiKey,
      ApplicationName = this.GetType().ToString()
  });
  
  UserCredential credential;
  using (var stream = new FileStream("secrets.json", FileMode.Open, FileAccess.Read))
  {
    credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(GoogleClientSecrets.Load(stream).Secrets,new[] { YouTubeService.Scope.YoutubeUpload },"user",CancellationToken.None);
  }

  youtubeService = new YouTubeService(new BaseClientService.Initializer()
  {
      HttpClientInitializer = credential,
      ApplicationName = Assembly.GetExecutingAssembly().GetName().Name
  });

  var video = new Video();
  video.Snippet = new VideoSnippet();
  video.Snippet.Title = "Squirrel detected";
  video.Snippet.Description = "You should have bought a squirrel";
  video.Snippet.Tags = new string[] { "squirrel" };
  video.Snippet.CategoryId = "22"; 
  video.Status = new VideoStatus
  {
      MadeForKids = false,
      SelfDeclaredMadeForKids = false
  };
  video.Status.PrivacyStatus = "private"; 
  var videosInsertRequest = youtubeService.Videos.Insert(video, "snippet,status", fileStream, "video/*");
  videosInsertRequest.ProgressChanged += videosInsertRequest_ProgressChanged;
  videosInsertRequest.ResponseReceived += videosInsertRequest_ResponseReceived;

  await videosInsertRequest.UploadAsync();
}
private void videosInsertRequest_ResponseReceived(Video obj)
{
  Console.WriteLine($"done");
}

private void videosInsertRequest_ProgressChanged(IUploadProgress obj)
{
  Console.WriteLine($"Bytes: {obj.BytesSent}| Status: {obj.Status}");
}

(secrets.json contains a client_id and client_secret for oauth) While uploading the ProgressChanged event gets fired a few times. Obj.Status cycles from Starting, Uploading to Completed. Which is also the Status when ResponseReceived is fired.

Because I had to switch to YouTube I also changed the workflow a bit. Now I already upload the video when posting the preview toot as private on YouTube. So I can verify the complete contents before an approval. And YouTube can process the video while it is waiting for approval. At the end I just switch from private to unlisted if it is approved or delete the Video if no approval has been made for a period of time.

Another Pitfall

YouTube does not allow using service accounts when uploading Videos. This was crucial for my application because this should run on a server. While the code above works I hit a dead end here. So back to Mastodon:

FfMpeg

To reduce the filesize I went to FFMpeg. I can compress Videos and - if required - even resize them. Because I do not want to call something on the console, I needed a wrapper. I found FfmpegCore which does exactly what I want:

public async Task<bool> ResizeVideo(string source, string target)
{
	return await FFMpegArguments
		.FromFileInput(source)
		.OutputToFile(target, true,
		options=>options
			.WithVideoCodec(VideoCodec.LibX265)
			.WithVideoFilters(filter=>filter
			.Scale(videoSize: FFMpegCore.Enums.VideoSize.FullHd)
			)
		.WithFastStart())
		.ProcessAsynchronously();
}

This tiny little piece of code takes any videofile, compresses it as h265 (using default quality settings) and resized it to FullHD (1080 height) maintaining the correvt aspect ratio.

That all works fine on my machine, but because I want to host this (like all my bots) on my Synology-NAS I had to dockerize the contents. I was in luck this time, but this also turned out to be quite easy:

RUN apt-get -y update && apt-get -y upgrade && apt-get install -y --no-install-recommends ffmpeg A single Line in the DockerFile and everything was working smoothly

Fixing the Api

The reduced Filesize did not solve the problem that Mastodon did not want to upload my file. So I digged a bit deeper into the Mastodon API. The media upload used by MastoNet seemed to marked as depricated (v1) exactly for the reason why I did not work in my project: It was meant for small preview images, not for Video files.

A V2-Version is available but not implemented by MastoNet. So I forked the packages from Github and added a V2-Version:

public async Task<Attachment> UploadMediaAsync(MediaDefinition media, string? description = null,
AttachmentFocusData? focus = null)
{
	media.ParamName = "file";
	var list = new List<MediaDefinition>() { media };
	var data = new Dictionary<string, string>();
	if (description != null)
	{
		data.Add("description", description);
	}

	if (focus != null)
	{
		data.Add("focus", $"{focus.X},{focus.Y}");
	}

	return await this.Post<Attachment>("/api/v2/media", data, list);
}

This was confusingly easy as I simpy just replaced “v1” by “v2”. Still it worked. Problem was: The next step, usually taking the returned Id and add this to a toot failed, because Mastodon was not finished, yet. I could simply have waited a long enough time, but decided to add a poll to receive status updates. Every Media of a known I can be requested. There is no statusflag about it processing but some properties give a hint: First there is the previewUrl this is filled at the very start after uploading. Tbe Url-Property, however isn’t. This will always be Null until everything is processed. So just requesting the Media every 10 seconds until the Url is filled did the trick;

public async Task<Attachment> WaitUntilMediaIsUploaded(Attachment attachment, int waitSeconds = 10, int maxWaitSeconds = 300)
{
	int totalTimeWasted = 0;
	while (attachment.Id != null && attachment.Url == null && attachment.PreviewUrl != null && totalTimeWasted < maxWaitSeconds)
	{
		Thread.Sleep(waitSeconds * 1000);
		totalTimeWasted += waitSeconds;
		attachment = await GetMedia(attachment.Id);
	}
	return attachment;
}

public async Task<Attachment> GetMedia(string mediaId)
{
	var attachment = await this.Get<Attachment>("/api/v1/media/" + mediaId);
	var prev = attachment.PreviewUrl;
	var url = attachment.Url;
	return attachment;
}

Finally

So that was the last Pitfall. Now everything works:

If a motion is detected my WebHook gets called
The Webhook posts a previewimage as a direct toot (only be seen by me) to Mastodon including a Preview Image
If “fav” the toot when this is a valid Squirrel detection
The Bot checks for new Favs and finds mine
The according Video is found in the LiteDB
The Video gets downloaded via SMB-Stuff
The Video gets Resized by FfMpeg
The resized video gets uploaded to Mastodon
A public toot containing the video is created

(Using standard Mastodon-Features, all toots will get deleted after two weeks reducing the space required)

But WHY?

I am very aware that this was mostly a waste of time. A lot of time for a simple “Lets show when there is a squirrel”. Simone Giertz once did a great TED - Talk Why you should make useless things. While she talks about robots, by fun thing to do was alway coding. No matter how useless the results are. So: Yes. I totally agree with her. Creating stupid things just because it is fun, with noone asking why this takes so long and even be able to quit if something becomes boring is a nice diversion to the normal coding reality I have for paying the rent.

I hope you also had some fun reading this; I had writing it :)

Two Apps at the same time: Deploy SharePoint Add-In to Cloud AND On-Premise

dotnet@oles.cloud (Ole Albers) — Tue, 13 Dec 2016 12:00:00 +0100

What the fuzz is this about?

I am not absolutely sure if I am the first one who did this, but it seems at least noone else wrote it down until now. :) We hat a customer who liked our Office365 Provider-Hosted App “Mydea”, but wanted us to install that on his SharePoint 2013 - Installation. In Germany a lot of companies do not trust the cloud. So we had three options:

Tell the customer this does not work
Copy&paste the whole solution and change it so it works on OnPrem
Do something else

The first option - of course - wasn’t valid for me. The second would have worked. But I did not want to end up in maintanance hell. Our product is about innovation management. And the most innovative about that is the product itself. We develop agile, adding new features (and sometimes bugs) continuously. We do not want to do that twice.

So I ended up on the third option: Changing my solution so it will run in an Office365 - SharePoint and on an On-Premise installation of SharePoint 2013 (or above).

Common sense about that path seems to be: That does not work. But it does. It just is a real bumby road to get there. This also means: You should know the basics. You should have created at least one Add-In for Office365 and one for SharePoint 2013 or above. I will not explain these steps here (which are complicated enough for local installation). I also cannot help you there in the comments or by mail.

There are plenty of documents out there that explain what’s needed for this. You should read those, first.

If you find this article useful and earn lots of money because of it: Feel free to buy me a beer :)

Analyzing the differences: What has to be done?

I started with an Office365-App and added the possibility to run that App under an OnPrem-Installation, too. But you should be able to go the other way with little modifications.

My first step was to create two dummy “Hello World"ish Applications. One for O365 and one for an OnPrem-Installation and look at the source, to “diff” it. These are the conclusions I came around:

At most of us use NuGet-Packages, that was the starting point. There are different Packages: You need the following:

OnPrem: - AppForSharePointWebToolKit

Office 365: - AppForSharePointOnlineWebToolKit - Microsoft.SharePointOnline.CSOM

These Packages result in different DLLs with the Prefix “Microsoft.SharePoint.Client.*”, but different Versions. OnPrem: - Version 15.xxx Office 365: - Version 16.xxx

The Sourcecode of the *.cs - files created by that NuGet-Packages (_SharePointContext.cs_ & _Tokenhelper.cs_) are exactly the same. So no need to change anything there.

Finally you usually store your Office365-Apps on Azure, but don’t for On-Premise-Installations. This results in different Authentications: OnPrem: - Using Certificates - Using Windows Authentication

Office 365: - Using ClientSecret - Using OAuth Authentication

These Informations are stored inside the Web.Config

Before we start

Create a new Branch in your repository. You never know what can go wrong :)

The Web-Project (MVC most likely)

Creating different Build configurations

The main purpose if this action was to have just one CodeBase for both versions. So new features or bugfixes have only to be made once. No extra code, no branch, no nothing.

So I started by creating new Build Configurations. You can create them using “Build - Configuration Manager”:

Create two Configurations. One for Debug and one for Release. Make sure to copy the Settings from the existing “Debug”- and “Release”-Configuration

At this point, only do this for your web project and - if you have - any backend-projects that are used by your webproject. We will add others later on, but this way you have the chance to validate every single step:

Now its time to create your config file. Find your web.config, click on it with the right mouse button and select “Add config transform”. After that you should have a Config file for each configuration

Nuget-References

I used the PnP - Packages to develop our O365-Add-In. If you only use the “base” Packages that come with the Add-In-Solution-Template you might have different packages. Nonetheless you will always have a package “AppForSharePointOnlineWebToolKit”.

Package-wise the main difference between the O365/OnPrem - road are the Azure- and OAuth-packages that are only on the 365-PnP. Also the Office365 Libraries use Version 16, While OnPrem use Version 15.

To be able to keep your application updated no matter what configuration you use, create two dummy projects e.g. dummy365 & dummyOnprem. It does not matter what kind of projects these are. The only purpose of these projects is that there are always the packages for both O365 and OnPrem in the ./packages - folder.

Add the O365-pnp-NuGet-packages to your dummy365 and the 2013-pnp-nuget-package to your dummyOnPrem

Now: Before you continue, **make a backup of you .csproj - file. It conains information that will be lost after the next step.

Now add the “other” PnP-Package. If you already have the SharePointPnPCoreOnline-Package installed. now install the SharePointPnpCore2013 NuGet-Package (and vice-versa)

csProj-Changes

You can now diff your csproj - files before and after. Or simply believe my words: There are now two differences. One of them is the OfficeDevPnP.Core - package. The HintPath from the O365 contains SharePointPnPCoreOnline, the HintPath from the 2013-Version contains SharePointPnpCore2013. This does not seem to be an issue as both DLLs are identical. Nonetheless we will have a look at it. Just in case.

The more interesting detail is the Reference to “Microsoft.Online.SharePoint.Client.Tenant”. The Office365 - Version uses Version 16.x and the OnPrem - Version uses 15.x. This is absolutely something we have to care about.

Luckily, the csProj allows some conditions. So create the following new lines:

<ItemGroup Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">
<Reference Include="Microsoft.Online.SharePoint.Client.Tenant, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCore2013.2.9.1611.0\\lib\\net45\\Microsoft.Online.SharePoint.Client.Tenant.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="OfficeDevPnP.Core, Version=2.9.1611.0, Culture=neutral, PublicKeyToken=3751622786b357c2, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCore2013.2.9.1611.0\\lib\\net45\\OfficeDevPnP.Core.dll</HintPath>
<Private>True</Private>
</Reference>
</ItemGroup>

<ItemGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
<Reference Include="Microsoft.Online.SharePoint.Client.Tenant, Version=16.1.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, processorArchitecture=MSIL">
<HintPath>..\\packages\\Microsoft.SharePointOnline.CSOM.16.1.5813.1200\\lib\\net45\\Microsoft.Online.SharePoint.Client.Tenant.dll</HintPath>
<Private>True</Private>
</Reference>

<Reference Include="OfficeDevPnP.Core, Version=2.9.1611.0, Culture=neutral, PublicKeyToken=3751622786b357c2, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCoreOnline.2.9.1611.0\\lib\\net45\\OfficeDevPnP.Core.dll</HintPath>
<Private>True</Private>
</Reference>
</ItemGroup>

Do not just copy/paste the content from here! Use the lines from within your csProj-File instead! Versions will change over time. And when you read these lines, my version used might already be outdated.

As a last step, remove the original lines from within the main ItemGroup containing these two libraries.

Well done. You should now try to rebuild your solution on all your four Build configurations, where the OnPrem-Configurations should create a Microsoft.Online.SharePoint.Client.Tenant.dll with Version 15 in your bin-directory and the Office365-Version should create a V16-dll. If you encounter a problem try to unload and reload the projects.

Please be advised: I am not sure what happens if you update one of those two Nuget-Packages. You might have to modify your csproj-file again

Web.Config

Now its time to modify your Web.Configs.

Add the following lines just before the closing "” section into your *.OnPrem.config - files:

<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly xdt:Transform="Replace" xdt:Locator="Condition(./\_defaultNamespace:assemblyIdentity/@name='Microsoft.Online.SharePoint.Client.Tenant')">
<assemblyIdentity name="Microsoft.Online.SharePoint.Client.Tenant" publicKeyToken="71e9bce111e9429c" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-15.0.0.0" newVersion="15.0.0.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>

As before: If at time of reading, the most current version is not 15.0, but 15.1 or higher replace the version numbers instead.

Now it is time to modify your “normal” Configs for Debug and Release

<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly xdt:Transform="Replace" xdt:Locator="Condition(./\_defaultNamespace:assemblyIdentity/@name='Microsoft.Online.SharePoint.Client.Tenant')">
<assemblyIdentity name="Microsoft.Online.SharePoint.Client.Tenant" publicKeyToken="71e9bce111e9429c" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-16.1.0.0" newVersion="16.1.0.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>

This is not 100% necessary at this time as these values should already be in the main Web.Config, but this makes sure it still works if that changes. Again: Check the version number.

Authentication

Your O365 - Config Files should still contain something like this:

<appSettings>
<add key="ClientValidationEnabled" value="true" />
<add key="UnobtrusiveJavaScriptEnabled" value="true" />
<add key="ClientId" value="aaaaaaaa-beef-face-bbbb-cchcchcchcch" />
<add key="ClientSecret" value="lsdschnabbeldiwuppsfkjashriuae=" />
</appSettings>
`now your OnPrem-configs looks a little bit different:`
<appSettings>
<add key="ClientValidationEnabled" value="true" />
<add key="UnobtrusiveJavaScriptEnabled" value="true" />
<add key="ClientId" value="aaaaaaaa-beed-face-bbbb-cchcchcchcch" />
<add key="ClientSigningCertificatePath" value="C:\\Certs\\myCert.pfx" />
<add key="ClientSigningCertificatePassword" value="p@ssw0rd" />
<add key="IssuerId" value="bbbbbbbb-beef-face-bbbb-cchcchcchcch" />
</appSettings>

The main difference here is that you don’t just use a ClientId/ClientSecret - Combination for an OnPrem-Installation but have to use a Certificate. I will not describe here how to obtain it. It is just the “normal” OnPrem-procedure for provider hosted apps. Also be aware that the IssuerId is not the ClientId.

If you need help on this part: There are plenty of documents about this. For example onMSDN

Backend, Webjobs and other projects that talk directly to SharePoint

App.Config

If you have any other Projects than the Web Project and the AddIn, you might have to do some extra steps. For example if you use a backend-project, you have to modify that, too. Ignore this chapter if you don’t have any additional projects.

But if you have any console application (like Azure WebJobs) you have to modify those configs, too. Sadly. Config-transform does not work out of the box with App.Config - files. But there is a way to change that.

First: Create your Config Files manually. So select Add - New Item and select “Application Config File”. Name it App.OnPrem.config. Now open the csProj-File with Notepad. Look for an ItemGroup containing the App.Config. There already should be an App.Debug.config and App.Release.config with a “DependentUpon” - Attribute. Just copy that attribute to your newly created files. So that you end with something like this:

<ItemGroup>
<None Include="App.config" />
<None Include="App.Debug.config">
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Debug.OnPrem.config" >
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Release.OnPrem.config" >
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Release.config">
<DependentUpon>App.config</DependentUpon>
</None>
</ItemGroup>

After that all your OnPrem-Configs should be displayed as children from App.Config in Visual Studio (2015). Now transform should work. You can check your csproj - File. If there is something like <UsingTask TaskName=“TransformXml” ../> in it you are ready to go. If not, you are probably using an older Visual Studio version. I would not advise it, but you can go with it, but have to do some extra steps: XDT Transforms for older VS Versions

Now its the same fun again with the config files as before. So add the same stuff into the App.*.config - files as you have done with the web.configs. In addtion add the transform Property to the configuration tag, so it looks like this:

Test what you have done until now

Now you should build your Solution for all four configurations. Check if all DLLs where created correctly for the O365 and OnPrem-Configurations. Deploy your application still with the O365 - Settings. Everything should look like before.

Changes on the Web Project

Now it is time to modify your main SharePoint-Project.

First of all: Add your SharePoint-Project to the build configurations. The good thing is: This time you don’t have to care about the configs. There will most likely be an app.config in your SharePoint-project, but you can just ignore it, because there are no real references in this project.

SharePoint - URL

The (nearly) final step is to put the right SharePoint-URL to your AddIn-Project depending on your Configuration. The simple way is to just enter the right URL each time you start debugging. The more complex (but better) way is to go the same way as you did with the MVC (Server) part. To do this, enter your Build-Configuration Settings again and set “Debug.OnPrem” and “Release.OnPrem” for your AddIn-Project.

The URL is stored in the user-settings instead of the project settings. But you can use the same magic to make a conditional path there. Open the file YourApp.csproj.user and add the condition to the “SharePointSiteUrl” - entry:

<PropertyGroup>
<SharePointSiteUrl Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">http://myDevMachine:10000/sites/Dev\_Ole/</SharePointSiteUrl>
<SharePointSiteUrl Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">https://my365AppName.sharepoint.com/sites/dev-ole/</SharePointSiteUrl>
</PropertyGroup>

IF you want you can also simply delete the user-file and put these values into your main .csproj. Sadly this cannot be loaded “on the fly” by switching your Build configuration . You have to unload and reload your project for this setting to be re-read. This is no issue when deploying your solution using an automated build (like tfs online), but while debugging locally.

Target office version & SharePoint-URL

The csproj - file contains an entry called “TargetOfficeVersion” which should contain the value “16.1”. Just like before we need to modify this using conditional values.

<TargetOfficeVersion Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">16.1</TargetOfficeVersion>
<TargetOfficeVersion Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">15.0</TargetOfficeVersion>

You can validate this setting while selecting the properties of the project and display the “Sharepoint”-Tab. Under “Target Sharepoint version” it should say “SharePoint 2013” if 15.0 is used or “SharePoint Online” if 16.1 is used.

Authentication

Office365 uses OAuth for Authentication. For SharePoint 2013 you have to use Windows Authentication instead. As I wrote before: There are a few documents out there explaining the Certificate-Part. But you also have to tell your project to use Windows Authentication. Again we use the same conditions like before, but this time use it for the IISExpressAuth:

<IISExpressWindowsAuthentication Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">enabled</IISExpressWindowsAuthentication>
<IISExpressWindowsAuthentication Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">disabled</IISExpressWindowsAuthentication>

AppManifest

The Appmanifest does not allow any conditional entries. That’s why we have to create a copy of it and do the conditional stuff inside the csproj file. So first create a copy called AppManifest.OnPrem.xmlns

In that file change the “SharePointMinVersion” from 16.0.0.0 to 15.0.0.0 in the “App"-Section.

Now it is finally time again to modify the csproj - File. You should be used to it by now :)

<ItemGroup Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">
<AppManifestFile Include="AppManifest.OnPrem.xml">
<SubType>Designer</SubType>
</AppManifestFile>
</ItemGroup>
<ItemGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
<AppManifestFile Include="AppManifest.xml">
<SubType>Designer</SubType>
</AppManifestFile>
</ItemGroup>

Things to mention on V 15.0 (SharePoint 2013)

While most of the code works on Version 15.0 as well on 16.1, there are some minor differences between these two. The following issues appeared on my project: - FieldUserValue has no Property “Email” - Microsoft.Client.SharePoint.Folder has no “Exists()” Both work on 16.1 but not on 15.0. So you have to write your own code to get around this issue.

Custom Compiler Symbols

Because of the differences between V16 and V15 you might want to have custom code depending on your configuration. Just like the DEBUG - Symbol you can add your custom Symbol for the OnPrem-Configurations. Just select the properties of each project you need that symbol in, select the Build - tab and into “_Conditional compilation symbol_s” enter “ONPREM” (or whatever you like):

Afterwards you can use the following code:

#if OnPrem
var userMail=GetMailFromUser(fieldUserValue); 
#else 
var userMail=FieldUserValue.Email;
#endif

Final thoughts

You are done now. Just keep in mind: If you use any Azure-connections (e.g. for caching, insights, blob-storage) you will have to handle that, too if you need to get rid of any external connection. The custom Compiler Symbols should help you with that task

FlashAirDownloader 2

dotnet@oles.cloud (Ole Albers) — Sun, 01 May 2016 12:00:00 +0200

⚠️ Warning

Development stopped! Please read this before downloading FAD

📝 Note

Keep your Version updated! Always click onto the “Check for Updates” in the Program after installation

The FlashAirDownloader 2 is out there (Because of my Laziness I will just call it FAD from now on)

A lot of people used the FlashAirDownloader which was meant as a personal tool and quick hack to access my Toshiba FlashAir Card. As more and more people asked me for features (and some for the code) I decided to make it open source so more folks can help improve the tool.

Sadly so: The code was a mess. As I told you: Just a quick hack. It would be to embarrassing to make this public :D So I decided to rewrite it from scratch: This time making it right regarding code quality and UI design. Nonetheless this program still uses Framework 4.0, cause I know there are Windows XP users out there who have no other possibilities to access the FlashAir.

But enough of talk. Here is the new FAD:

You can download the source directly from Github. Feel free to make the code better. Or download the binary here:

(The download from the old site is gone, please build from source.)

You will get an annoying amount of warnings from your browser and/or operating system. This is because the file isn’t signed. Valid signing requires a valid certificate which does not come for free. (and requires some amount of work) Feel free to build from source in GitHub if you have any doubts.

First start

After Installation and starting FAD you should see something like this:

If your FlashAir - card is already configured it should be found and the display should change to this:

For now: Keep calm and do not click on any of the blue squares. Use “File/Settings” instead.

Setup

Program Settings

Check if your settings are ok. Here is a small description from the settings:

FlashAir Url This is the Url from your FlashAir. You can try if you can access your FlashAir by entering that address into your Browser
Local Path Here the Images will be saved when using the Auto-Mode
Existing Files (Auto mode) Decide what to do if the file already exists.
Delete files after downloading (Auto mode) If checked, the files will be deleted after a successful download from the FlashAir
Flashair Imagepath Where your Images are located (may contain subfolders)
Filetypes to copy (Auto mode) Decide if you only want image files or other files, too
Load Thumbnails If enabled Thumbnails from Images on the FlashAir will be collected and shown. Might take a while. Thumbnails for local files are always displayed regardless of this setting.
Automatic folder Creation (Auto Mode) * Multiple Cards Use your Card ID or AppID to create subfolders if you use more than one card * Create By Date Decide in what format the directories should be named * Custom You can als create your own custom path. {0} stands for the CardId, {1} for the AppId and {2} for the current date. Examples about the date-format can be found here
Styling Decide if (and what) images should be displayed on the start page and the settings page of FAD.

Don’t forget to click onto [Save] under Card Settings in the lower left area of the screen.

Card Settings

Now after the application is set up it is time to check your FlashAir-Card. Just put your SD-Card into the computer and click onto [Load] in the upper left corner. After a few seconds three new tabs should appear: “Vendor”, “WLANSD” and “Disable Functions”

Click onto “Vendor” to setup the most important settings. I will only describe the most critical values. A complete list is available at the Toshiba FlashAir Website.

First make sure the Connection Timeout is correct. It must be at least 1 Minute. You can also set it to “0” to disable timeout completely. “App-Info” and “Card ID” can be modified if you want to use those values in the automatic folder creation.

If your Network settings already work, don’t touch them here. Otherwise follow the instructions on the FlashAir website.

Now make sure the “Upload Directory” fits the “Flashair Imagepath” in the “Application Settings”.

The last setting to check is “Upload Enabled”. This is required if you want to be able to save files onto the Flashair or want to delete files after downloading.

Now all you have to do is to click [Save] in the upper Left region and FAD is ready to go.

For the first run I would advice just keeping your card in your card-reader. It works exactly there like in your camera, but you can modify your settings more easy this way.

Auto - Mode

After FAD found your card click onto “Auto Mode” to start automatic downloading of your data:

At first the files are loaded recursively from your card. In the next step (if activated) the thumbnails gets downloaded. Then the files are being downloaded and deleted (if selected).

If the thumbnails cannot be retrieved red squares are displayed instead.

File Explorer

The file explorer does not download automatically images from the Flashair. Instead you can browse through your card and copy files manually. Just mark them by clicking and select the [«] or [»] - Button to copy them#

Known issues

https://www.youtube.com/watch?v=PtXtIivRRKQ

The FlashAir is sometimes a bit unstable (at least mine). There are a lot of reasons why something goes wrong. That’s why FAD tries up to five times to download or upload data. (Just Thumbnails are only tried once). There are many reasons why a download does not work. There can be a timeout, the Card can “crash” (it is a webserver running on it) or the camera just goes into standby. So often the best solution is to try again and or restart the camera and re-enter the FlashAir.

Any more questions?

Just look at the documentation on GitHub or open an issue there if you encounter any problems.

Love it?

Feel free to send some money to my Panama-Account

SCRUM and Git: Optimized Gitflow // with a lot of fancy colors :)

dotnet@oles.cloud (Ole Albers) — Wed, 04 Mar 2015 12:00:00 +0100

For a lot of teams, GIT is the first choice for maintaining the code. So it is for mine. But this post isn’t limited to GIT and should work with SVN or TFS with little modifications, too.

Why is the Git flow important when using SCRUM?

Aren’t those just two worlds, anyway?

Well. Kind of. But while SCRUM is quite flexible, scrums paradigm still has some very basic contents when it comes to manage projects:

You and your team work in sprints.
Inside these sprints there are user-stories you want to implement.

We decided to map sprints and user stories to our git flow.

Classic git flow:

You usually have the following structure in a git flow: (More fancy description and image at Atlassian)

You always got your master branch. From that branch you create single branches for each Hotfix. Your daily work is done in feature branches that start at the development branch. When a feature is finished, you merge it into your development branch. For a release, the development branch (which contains the added features) is the root of a new Release-branch. That release is tested and maybe modified until it fits the needs of your customer. When you are happy with that release you merge it back into the development branch as well as the master branch.

“Scrumified” Git Flow:

When trying to put SCRUM into the flow you keep your basic structure but add two additional layers to it. The single features are not created directly from the development branch. Instead at first for each sprint a new branch is created from the development branch. From the sprint there are branches created from each user story. After you’ve done that, the features are created from the corresponding stories. Merging back goes the same way. You merge your features back into the user story and the user story back into the sprint before merging the sprint back into your development branch.

This is a bit more work (and might look a little bit chaotic on these pictures at first) but it offers the advantages you have from “normal” git flow also to your sprint. Like before you only merge features that are finished. But now you also only merge user stories that are complete. And as a small bonus you can always analyze which user story was started and finished in which sprint. A Win-Win-situation like those unlikable marketing guys would call it. (If they were able to understand a single word I just wrote here) :)

IMAP-HTML-Autoresponder with PHP

dotnet@oles.cloud (Ole Albers) — Fri, 13 Feb 2015 12:00:00 +0100

Everyone knows spam. Everyone hates spam. And everyone who has his email-address written on the net (for example in the NIC-records for a domain) gets a lot of spam.

I am one of those guys who gets a lot of spam. Of course there are anti-spam solutions out there, but the best would be to not even receive spam. And spam filters need a lot of work and / or do not filter 100% perfect. That is why I decided to use a public email that has to go through a “gatekeeper”. A PHP-script that tells everyone that this is not my real mail address, but that there is another one where I could be contacted.

That email-address is not displayed in plain text but as an image to get rid of all those bots. SOME bots however should be able to get through: For example my hoster who needs to validate my email periodically. So these messages just are being forwarded.

I wanted to be as simple as possible. That’s why this script does not need much more than plain php. Only a replacement for phps “mail” was needed.

First step: Download swiftmailer. Just put the contents of the “lib” - directory onto your web space. That’s it. Ready to go.

Simple include the following line to access swiftmailer:

require\_once './swift/lib/swift\_required.php'; To access your mail you need to enter the IMAP-Settings. POP would work, too, but IMAP would be much more useful because we do not want to download the email-body of the messages processed:

// IMAP-Settings:
 $hostname = "{imap.hostname.com:993/imap/ssl}INBOX";
 $username = "username";
 $password = "password";
`Now enter your email-addresses. $targetMail is your real ("hidden") email-address where valid emails should be forwarded to. $thisMail is the public address that receives the spam.`
$targetMail="private@mydomain.com"; 
$thisMail="webmaster@mydomain.com";

Next you need to create an image with your favourite picture editing tool (GiMP, Paint.NET, PaintShopPro, etc.) This image should only contain your private email. Take a font you like. You might also use a “Captcha”-font. I - personally - hate that… :)

Upload that image to your web server and enter the url of that image. PNG would be a good choice.

$image=“http://www.example.com/images/mail.png";

Next, enter the text the sender should receive. By default everyone receives $templateHtml. For those who do not have the capabilities to display html (or don’t want to) a simple text-mail ($templatePlain) will be displayed. Users with a html-browser will see your email-image directly in the mail. Plain-text mail-clienst will display a link to the image instead.

$templateForward defines the mail content YOU will see if a mail is forwarded to you from a valid sender

So it is time to be creative :)

$templateHtml="<html><body>Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!<br/>But don't be afraid, there is a replacement for that. Just send your mail to: <br/><img src=\\"\_\_EMBED\_\_\\"/><br/><br/>If you cannot see that image: Click onto the following link to see my: <a href=\\"\_\_URL\_\_\\">new mail-address</a><br/>And no: This mail will not be forwarded. No one will see this until you forward it to my new mail-address. Thanks!</body></html>";

$templatePlain="Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!\\r\\nBut don't be afraid, there is a replacement for that. Just click onto the following link to see my: new mail-address: \_\_URL\_\_";

$templateForward="new message redirected";

As you can see there are two special tags in those templates. “__EMBED__” will be replaced with the embedded image (html-template only) and “__URL__” will contain a link to that image.

The last variable you can define is the white-list. Everything entered there is interpreted as a valid sender. Enter multiple values seperated by commas. $whitelist = "uberspace,denic.de,domainbox.net,romrobot.com,inwx.de,millerntor.hamburg,dotnet.work"; that’s all for configuration. Now just some explanation what the script does:

First of all the script checks your IMAP-Server for unread messages. It collects the header of each of those messages and compares the sender with the white-list. If the sender is recognized, the mail body is downloaded and sent as an attachment to the mail address you defined as $targetMail.

If the sender is not in the white-list, an automatic reply is generated using your html-template. It contains the image you downloaded as an attachment and embedded image of the mail. The last step is to mark the mail as read so it won’t be processed again.

In a perfect world…

Everything works now. But you can (and will) get problems sending the mail to the sender if the sender’s address does not comply with RFC 2822. While most spammers DO comply, mom DOESN’T… :) That’s because there is a mail client from a tiny little software company in Redmond, where they do not know much about standards. But this is no big problem at all. Just open lib\classes\swift\Mime\Headers\MailboxHeader.php from swiftmailer and put /* */ around the throw-part of the last function:

private function \_assertValidAddress($address)
{
   if (!preg\_match('/^'.$this->getGrammar()->getDefinition('addr-spec').'$/D', $address)) {
     /\* throw new Swift\_RfcComplianceException('Address in mailbox given \['.$address.'\] does not comply with RFC 2822, 3.6.2.');\*/
   }
}
`So finally here is the complete script.`
<?php
   require\_once './swift/lib/swift\_required.php';// IMAP-Settings:
   $hostname = "{imap.example.com:993/imap/ssl}INBOX";
   $username = "username";
   $password = "password"; // Script-Settings:
   $targetMail="private@example.com"; // "hidden" target mail-address
   $thisMail="webmaster@example.com"; // "official" mail-address (which receives the spam) 
   $image="http://www.dotnet.work/reply/mail.png"; // Image to be shown
   $templateHtml="<html><body>Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!<br/>But don't be afraid, there is a replacement for that. Just send your mail to: <br/><img src=\\"\_\_EMBED\_\_\\"/><br/><br/>If you cannot see that image: Click onto the following link to see my: <a href=\\"\_\_URL\_\_\\">new mail-address</a><br/>And no: This mail will not be forwarded. No one will see this until you forward it to my new mail-address. Thanks!</body></html>";
   $templatePlain="Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!\\r\\nBut don't be afraid, there is a replacement for that. Just click onto the following link to see my: new mail-address: \_\_URL\_\_";
   $templateForward="new message redirected";

   $whitelist = "uberspace,denic.de,domainbox.net,romrobot.com,inwx.de,millerntor.hamburg,dotnet.work";
 
   $allowedSenders=explode(",",$whitelist);
   $inbox = imap\_open($hostname,$username,$password) or die("Sorry, connection failed");
   $emails = imap\_search($inbox,"UNSEEN");
   $count=0; 
   echo "<pre>"; 
   if($emails) {
     $transport = Swift\_MailTransport::newInstance();
     foreach($emails as $email) {
        $count++;
        $header = imap\_fetch\_overview($inbox,$email,0); 
        $subject=$header\[0\]->subject;
        $sender=$header\[0\]->from;

        $messageHeader =  imap\_fetchheader($inbox, 1)
        if (strpos($messageHeader,"X-RESPONDERDATE")!==false) {
          continue; // no wild bouncing, please
        }

        if (strpos($sender,$targetMail!==false || strpos($sender,$thisMail!==false) {
           // We want no loops... :)
           continue;
        }
        echo "retrieving ".$subject." from ".$sender."\\r\\n";
        // Whitelist: 
        $isWhiteListed=false;
      
        foreach($allowedSenders as $allowedSender) {
           if (stripos($sender,$allowedSender)!==false) {
              $isWhiteListed=true;
              break;
           }
        }
 
        if ($isWhiteListed) {
           $email\_body = imap\_body($inbox, $email);
           $mailer = Swift\_Mailer::newInstance($transport);
           $message = Swift\_Message::newInstance()
              ->setSubject("Fw:".$subject)
              ->setFrom($sender)
              ->setTo($targetMail)
              ->setBody($templateForward)
           ;
 
           $attachment = Swift\_Attachment::newInstance($email\_body, 'origin.eml', 'message/rfc822'); 
           $message->attach($attachment);
 
           $result = $mailer->send($message);
           echo "whitelist redirect\\r\\n";
        } else { 
           $mailer = Swift\_Mailer::newInstance($transport);
           $message = Swift\_Message::newInstance();
           $swiftheaders = $message->getHeaders();
           $swiftheaders->addTextHeader('X-RESPONDERDATE', date('Y-m-d H:i:s'));
           $cid = $message->embed(Swift\_Image::fromPath($image));
           $html=str\_replace("\_\_URL\_\_",$image,$templateHtml);
           $html=str\_replace("\_\_EMBED\_\_",$cid,$html);
           $plain=str\_replace("\_\_URL\_\_",$image,$templatePlain);
           $message
              ->setSubject("Re:".$subject)
              ->setFrom($thisMail)
              ->setTo($sender) 
              ->setBody($html, 'text/html')
              ->addPart($plain, 'text/plain')
           ;
           $result = $mailer->send($message);
           echo "answered\\r\\n";
        } 
        imap\_setflag\_full($inbox, imap\_uid($inbox,$email), "\\\\Seen \\\\Flagged", ST\_UID);
     } 
  }
  imap\_close($inbox);
  echo "done redirecting ".$count." mails\\r\\n";
  echo "</pre>";
?>
`Just save this program as "reply.php", send test-mails and open "reply.php" in your browser to test the settings. When everything works fine you can simply add a cronjob so this script is run automaticly. The following example runs the script every 5 minutes:`
\*/5 \* \* \* \* php /var/www/reply.php

Just use the comments if you need any help

FlashAir Downloader for Windows

dotnet@oles.cloud (Ole Albers) — Thu, 25 Sep 2014 12:00:00 +0200

⚠️ Warning

Development stopped! Please read this before downloading FAD

📝 Note

**New Version available!**This version is obsolete. I would strongly suggest to use the FAD2 instead. You can download it here: http://dotnet.work/fad2.

Toshibas FlashAir is an SD-Card with built-in WiFi. It allows you to download Pictures from your DigiCam through WiFi. Sadly there is no working Windows-Application.

The FlashAir-Downloader solves that issue. It allowes Bulk-Downloading of your images. Your pictures can be saved in folders created automaticly containing date-information.

Installation:

Unpack the Zip and start the Setup.Exe

Start the Application and go to Settings:

Adjust the Settings if needed. Especially for “local Path”.

To use the Delete-Option you have to adjust the configuration of the SD-Card. Insert the SD-Card into your computer and open the file “CONFIG” file in the hidden “SD_WLAN”-folder with a text editor. Add the line UPLOAD=1 to the CONFIG-File.

Thats it. Close that window, turn on your camera and click onto “CONNECT”. If the connection works, you should see the folders and images of your SD-Card. Enter a folder or preview an image by double-clicking onto the icon. Mark the images you want to download and start downloading by clicking onto “Download”.

I give no guarantee about anything. This application shouldn’t do anything bad. But if your Camera explodes: Don’t blame me.This tool is and always will be free. But I don’t mind donations, of course :)

— Update V 1.1 —

I just updated the FlashAirDownloader to Version 1.1.

The following has changed:

Bugfix: If the file date is quite old (<2000) it could not be downloaded. This has been fixed. [*Yeah! Year 2000 bug in 2014. Well done, Toshiba *🙄 😁 ]
New Feature: You can now change the contents of the CONFIG – File directly. Just insert the SD-Card into your computer and Select “Card Properties
New Feature: If you use more than just One FlashAir-Card you can now automatically create subfolders for each card. You can use the internal “CID” ID for that or the Application ID which may contain any custom name.

(Tested with Windows 7 and Windows 8.1)

A new Version is available: http://dotnet.work/fad2

Emptying SharePoint-List the (quite) fast way

dotnet@oles.cloud (Ole Albers) — Fri, 10 Aug 2012 12:00:00 +0200

I already offered a solution to empty a SharePoint list at light speed

But this solution hast two big disadvantages:

You cannot empty a list that has more entries thatn the allowed list view threshold (default is 5000)
All Lookups will be lost

This solution has non of those drawbacks. It is much slower than that solution but also much faster than “normal” deletion of SPListItems or using a Caml-Query to do that.

Just be asure that this is not thread-safe. So make sure noone is writing into that list in the same time or that lines might not be deleted.

private int rest;
public static void TruncateList(Guid listId, int limit)
{
SPSecurity.RunWithElevatedPrivileges(delegate()
{
using (SPSite site = new SPSite(SPContext.Current.Site.ID))
{
using (SPWeb web = SPContext.Current.Web)
{
try
{
\_rest = limit;
web.AllowUnsafeUpdates = true;
SPList listCopy = web.Lists\[listId\];

ContentIterator iterator = new ContentIterator();
iterator.ProcessListItems(listCopy, ProcessItem, ProcessError);

web.Update();
web.AllowUnsafeUpdates = false;
}
catch (Exception ex)
{
if (!ex.Message.Contains("Limit reached"))
{
Debug.WriteLine(ex);
}

}
}
}
}
);
}

public static bool ProcessError(SPListItem item, Exception ex)
{
if (!ex.Message.Contains("Limit reached"))
{
Debug.WriteLine(ex);
}

return true;
}

public static void ProcessItem(SPListItem item)
{
if (rest<=0)
{
Debug.WriteLine("Limit reached");
throw new Exception("Limit reached");
return;
}
item.Delete();
\_rest–;

}

Emptying SharePoint lists the ultra fast way

dotnet@oles.cloud (Ole Albers) — Thu, 09 Aug 2012 12:00:00 +0200

SharePoint is damn slow when deleting lines from lists. Each entry can take up to some seconds which isn’t nice when you have to delete thousands of lines.

Also no Trunc-Methode is available.

Other solutions found on the net creating xml-files to bulk delete aren’t very fast either.

That’s why I decided to write my own fast truncate-method for SharePoint-Lists.

All content of that list will be deleted. Use at your own risk!

What it does:

First a template of that list is being created. After that the list is deleted completely. Than a new list using that template is being created and at the end that template is being deleted.

Works damn fast (just a few seconds even on big lists). IF anything goes wrong you can use the template to create a list manually.

public void TruncateList(Guid listID)
{
SPSecurity.RunWithElevatedPrivileges(delegate()
{
using (SPSite site = new SPSite(SPContext.Current.Site.ID))
{
using (SPWeb web = SPContext.Current.Web)
{
try
{
web.AllowUnsafeUpdates = true;
// alte Werte merken
SPList listCopy = web.Lists\[listID\];
string title = listCopy.Title;
string description = listCopy.Description;

string backupName = string.Format("BACKUP{0}{1:d} {1:HH}{1:mm}", title,DateTime.Now);

// save template
listCopy.SaveAsTemplate(backupName, backupName, string.Empty, false);
SPListTemplate template = web.Site.GetCustomListTemplates(web)\[backupName\];

// delete list
listCopy.Delete();
web.Lists.Add(title, description, template);

//  remove template
SPList gallery = web.Lists\["List Template Gallery"\];
foreach (SPListItem item in gallery.Items.Cast().Where(item => item.Title == backupName))
{
item.Delete();
break;
}

web.Update();
web.AllowUnsafeUpdates = false;
}
catch (Exception ex)
{
\_log.Error(ex);
throw;
}
}
}
}
);
}