Any IMAP experts out there?

[RogerS]

I access my email via IMAP using either Apple Mail on the desktop or K9-Mail on the Android phone. All works fine apart from one niggle that I don't understand.

I set up Rules on Apple Mail to move incoming emails to specific folders. For example, email notifications from UKW from threads I have subscribed to, are moved (filtered, if you prefer) to a folder called UKW. Then it is a simple matter to periodically Select All and delete. So far, so good.

However, I noticed that these notification emails do not appear in the Inbox on K9. Digging in to the actual mail on the server using their webmail interface shows that those moved emails have been marked as 'Marked as deleted'. I presume that Apple Mail gets in ahead of K9 - moves the email and this results in the email being flagged and thus invisible to K9.

Temporarily turning off the rule proves that (a) notification emails are no longer moved ...as one would expect (b) notification emails become visible on K9 (c) they are not 'marked as deleted'.

Why does moving the email result in 'marked as deleted' ?

Second question. If I go into the inbox folder using webmail and change the email as Undeleted - it shows up there in webmail as a normal email but I cannot see it - either in K9 or in Mail. Is this because there is some hidden syncing mechanism and that the mail clients say to themselves (a) I've already got this email and moved it to a new folder (Apple Mail) and (b) I've already looked at this and it was deleted so I'm not showing it (K9) ?

TIA

 

[Eric The Viking]

I don't like IMAP, but I like K9 even less.

I've recently given up email on the Android devices because it's so unreliable. And K9 has as much user-friendliness as an annoyed porcupine.

Regarding your questions: move = (copy + delete) in UNIX land. It may be that's what's occurring, and the server isn't doing it correctly (who's your new email service provider?).

Q2: "Undeleted" will still leave it marked as read - could you have a filter to display only unread messages somewhere?

It pains me to say so, but of all the email accounts I've recently had, the only two that have behaved properly have been MS Exchange for work (me using IMAP to get at it), and Google's gmail. Gmail is cheap to use, so you might try setting a gmail account up and testing for those problems. Also there are several other IMAP clients for android that work well with gmail, including Google's own. As I said, I fine K9 infuriating, especially as some of its routing rules half work or have quite perverted logic.

I don't think you'll nail it conclusively unless you do some systematic testing.

E.

 

[RogerS]

123-reg is my hosting company for my domain/email address.

I quite like K9 - better than the native mail app (for reasons that escape me at the moment but there was a good one for switching IIRC)

 

[defsdoor]

Move doesnt equal copy and delete in unix-land unless you are moving between file systems. Move = Move.

On a filesystem level (lets assume the IMAP server is storing emails as individual files, just as dovecot does with Maildir storage) moving it from one mail box to another (or at fileystem level from one directory to another) results in a removal of the directory entry from the original directory contents list (a simple linked list stored on the disk) and an insertion of a new directory entry in the new directory's contents list. The file itself doesn't get touched (unless the directories are on different filesystems.)

Files themselves are managed by inodes and inode trees. Directories point to inodes. The good thing about this is that you can have multiple directory entries pointing to the same inodes/files (hardlinks). Files are deleted when the reference count on the inode is reduced to 0 (no more directory references to it).

I suspect what is happening here (now at IMAP level) is that the email existed in the IMAP folder's index (how IMAP keeps track of the emails and allows searching). When you move it elsewhere, this implementation, instead of simply removing the index entry, keeps it but marks it as deleted (although this opens up the question of when is the deleted entry purged)

It could just be that your IMAP client has seen the email and now it isn't there and it is it's local index still showing the item as deleted.

 

[defsdoor]

123 Reg's IMAP server certainly looks like dovecot - they've bespoked some responses but others are identical -

My IMAP server

* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN
0 OK Pre-login capabilities listed, post-login capabilities have more.

123 Reg's

* CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS UIDPLUS LIST-EXTENDED I18NLEVEL=1 AUTH=PLAIN
0 OK Pre-login capabilities listed, post-login capabilities have more.

 

[RogerS]

Very helpful thanks, defsdoor.

My use of the term folder is an inaccurate one although conceptual true. It is on Apple Mail that the individual mailboxes are set up. You can see them in Finder as distinct directory folders. The email is physically moved from the main Inbox mailbox to the 'rule' mailbox. So if this does constitute a move (ie copy and delete) then it would explain why the email on the IMAP server gets 'marked as deleted' - otherwise I can't see how it gets marked.

There are no 'rule' mailboxes set up on the IMAP server - for clarification.

You mention searching on the IMAP server. How does one do that? Or why would one want to? Searching is done locally on the Mac on the available mailboxes.

Am I correct in saying that the IMAP server 'knows' about individual email clients and maintains a 'read' list for each one. This 'read' list tells the IMAP server which emails to send down to the email client the next time the latter is online and polling the server? So an email 'marked as deleted' and polled by, say, K9 would not appear in K9's inbox because it's been 'virtually' deleted. But even if one marked it as 'undeleted', the very fact that K9 has polled the server and 'ticked the box' as it were on the servers' 'read' list means that the server won;t send the email down to K9. Is that how it works?

 

[defsdoor]

With IMAP everything that goes on on the client is purely for your local convenience. For example your client might replicate all your subscribed folders so that they are available immediately, allow searching sorting etc..

But all of this functionality is also available remotely by talking to the IMAP server. Clients that cannot keep a local copy of the entire mailbox (phones for example) still allow you to search etc.. and this is done by asking the server to search and getting the results back.

Emails are marked as read on the IMAP server once and once only and all your IMAP clients will show this single read status (I demo the pros of IMAP by opening the mailbox in webmail, thunderbird (for example) and on a phone and they all in real time reflect the change of an email when one of the clients opens it (marks it read).

The keeping track of what has changed and how a client gets up to date with the changes in between connections (clients that stay connected with IMAP IDLE get real time change notifications) is done by the client asking for a summary list of emails and working out what has changed and what might need the entire header and body fetching etc..

 

[defsdoor]

Sorry I forgot to clarify - the IMAP server doesn't know one client from another - it doesn't keep track of where the client is. The client has to do that by listing. There are various ways to do this depending on the CAPABILITY list the server issues when asked.

 

[RogerS]

Many thanks for the extra info.

So is my assumption how/why the emails get 'marked as deleted' following a move of the email to a new mailbox on the Mac is correct?

 

[defsdoor]

If the mailbox is a different account or location outside of the IMAP folder tree then yes

 

[RogerS]

Brilliant, thanks. Once you know the right words to Google ......IMAP folder tree!

https://discussions.apple.com/thread/5119594

All is explained and confirmed.

 

[Eric The Viking]

So that probably counts as "moving" between filesystems then ;-)

(Yes, I know... DB record, not an inode, etc...)

E.

 

[toysandboats]

I'm no expert but did slowly get disenchanted with K9. Finally swapped to AquaMail and have now used it happily for almost a year.
David

 

[RogerS]

I forgot to mention one serious drawback to using an IMAP mail server and that is, unless you have deliberately copied your messages into a program or app on your own device then when/if you move away from your current IMAP provider (as I did) then all your emails (in and out) go pffft! and disappear in a puff of smoke. DAMHIKT