"They came in the dead of the night. I woke up and saw armed men docking off from boats with long knives. I've never seen them before in my life," says Ben Ahmid.

Together with 35 members of his family, Ahmid fled the sleepy village of Kampung Tanduo with only his clothes on his back. He said they had to leave everything behind: the home he shared with his family for decades and his livelihood.

Now he says all he wants is his old life and that of his family back.

But no one knows when that will happen. The new occupiers of Kampung Tanduo say they have no plans of leaving. 

Calling themselves the Royal Sulu Army, they demand recognition and the right to permanently stay in Sabah. There are about 500 of them, they say, and are followers of the Sultanates of Sulu, who once ruled over Sabah and Northern Borneo. 

Sultan Raja Muda Kiram, the leader of the group, says he and his clans are the descendants of the great and brave Tausug warriors who once ruled Sabah. He says he intends to reinstate that lost glory.

The claim to retake Sabah was officially formalised by the Philippine government in the 1960s but it has been dormant since. Malaysia until now pays annual royalties to the Sultanates.

Malaysian forces say they have already cornered the group and imposed a deadline for it to head back to the Philippines voluntarily.

To see what the situation is really like on the ground, we travelled to Lahad Datu. We rented a small pump boat and headed out to film by the coast.

As we travelled further east, I noticed a small boat sailing parallel towards the same directions as ours. It moved in closer very quickly. By then we realised it was the Malaysian navy so we stopped the boat immediately. 

"You're not allowed to sail here," we were told. In a few minutes, another Malaysian patrol boat with eight soldiers arrived. "No need for you to come here, all is well. There is peace here," they told us.

We were asked to go to a police station, where I was the first to be interrogated.

"You're a woman, what are you doing on a boat with six men? It's dangerous!" an officer asked.

I simply told him I am a journalist, but I was told I could be part of the Sulu Army. The ordeal lasted several hours, and my colleagues were subjected to a similar grilling.

Our troubles were testament to the existing paranoia over Sabah's security. Many Sabahans worry they may come under attack by the Royal Sulu Army group anytime. "They are amongst us, they've been here the whole time," one said.

Filipinos we've spoken to here say they have often been regarded with suspicion. But such discrimination is something they've got used to over the years. Life in Sabah, they say, is still better than what they had back home.

There are hundreds of thousands of Filipinos in Sabah. Most of them came here to escape poverty and militarisation in southern Philippines. But even after generations of living here, many of them have remained poor and stateless.

Ben Ahmid says all he wants is to to live his life in peace. Along with about 80 others who fled the standoff, they would like to see an end to the conflict that has little to do with them.